[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:24:55 dist-test-slave-ffrd krb5kdc[12933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:24:55 dist-test-slave-ffrd krb5kdc[12933](info): set up 2 sockets
Aug 10 07:24:55 dist-test-slave-ffrd krb5kdc[12933](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 07:24:56.316162 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.298s user 0.004s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:24:56 dist-test-slave-ffrd krb5kdc[12933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:24:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:24:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:24:56.473357 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:40833
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:42335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:40833
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:56.589892 12956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:56.590167 12956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:56.590210 12956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:56.594249 12956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:24:56.594326 12956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:56.594345 12956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:56.594362 12956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:24:56.594378 12956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:24:56.599745 12956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:42335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:40833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:40833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:24:56.601070 12956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:56.602162 12956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:56.612298 12969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:56.612326 12967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:56.612526 12964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:56.612663 12956 server_base.cc:1034] running on GCE node
I20260810 07:24:56.613371 12956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:56.614542 12956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:56.615757 12956 hybrid_clock.cc:648] HybridClock initialized: now 1786346696615697 us; error 97 us; skew 500 ppm
Aug 10 07:24:56 dist-test-slave-ffrd krb5kdc[12933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346696, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:56.618917 12956 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:24:56.619961 12956 webserver.cc:460] Webserver started at http://127.12.157.254:36875/ using document root <none> and password file <none>
I20260810 07:24:56.620551 12956 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:56.620647 12956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:56.620921 12956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:56.623220 12956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "688ef6aca9dd43888d908e1dffcaacd9"
format_stamp: "Formatted at 2026-08-10 07:24:56 on dist-test-slave-ffrd"
server_key: "c2037740d5b7888574dc02c40b6715f2"
server_key_iv: "8709cbb294384f39c688364aee53c742"
server_key_version: "encryptionkey@0"
I20260810 07:24:56.623852 12956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "688ef6aca9dd43888d908e1dffcaacd9"
format_stamp: "Formatted at 2026-08-10 07:24:56 on dist-test-slave-ffrd"
server_key: "c2037740d5b7888574dc02c40b6715f2"
server_key_iv: "8709cbb294384f39c688364aee53c742"
server_key_version: "encryptionkey@0"
I20260810 07:24:56.628132 12956 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 07:24:56.631168 12979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:56.632555 12956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 07:24:56.632714 12956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "688ef6aca9dd43888d908e1dffcaacd9"
format_stamp: "Formatted at 2026-08-10 07:24:56 on dist-test-slave-ffrd"
server_key: "c2037740d5b7888574dc02c40b6715f2"
server_key_iv: "8709cbb294384f39c688364aee53c742"
server_key_version: "encryptionkey@0"
I20260810 07:24:56.632884 12956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:56.646576 12956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:56.650605 12956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:56.650841 12956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:56.658881 12956 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:40833
I20260810 07:24:56.658911 13069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:40833 every 8 connection(s)
I20260810 07:24:56.659907 12956 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:24:56.661741 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 12956
I20260810 07:24:56.661934 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:24:56.663498 13070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:56.663627 12919 external_mini_cluster.cc:1428] Setting key e8295d6aff9da2af5ef628ee214d3fd8
I20260810 07:24:56.671187 13070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 688ef6aca9dd43888d908e1dffcaacd9: Bootstrap starting.
I20260810 07:24:56.674032 13070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 688ef6aca9dd43888d908e1dffcaacd9: Neither blocks nor log segments found. Creating new log.
I20260810 07:24:56.675034 13070 log.cc:824] T 00000000000000000000000000000000 P 688ef6aca9dd43888d908e1dffcaacd9: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:56.677613 13070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 688ef6aca9dd43888d908e1dffcaacd9: No bootstrap required, opened a new log
I20260810 07:24:56.682451 13070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 688ef6aca9dd43888d908e1dffcaacd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688ef6aca9dd43888d908e1dffcaacd9" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 40833 } }
I20260810 07:24:56.682756 13070 raft_consensus.cc:374] T 00000000000000000000000000000000 P 688ef6aca9dd43888d908e1dffcaacd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:56.682829 13070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 688ef6aca9dd43888d908e1dffcaacd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 688ef6aca9dd43888d908e1dffcaacd9, State: Initialized, Role: FOLLOWER
I20260810 07:24:56.683384 13070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 688ef6aca9dd43888d908e1dffcaacd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688ef6aca9dd43888d908e1dffcaacd9" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 40833 } }
I20260810 07:24:56.683568 13070 raft_consensus.cc:388] T 00000000000000000000000000000000 P 688ef6aca9dd43888d908e1dffcaacd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:24:56.683653 13070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 688ef6aca9dd43888d908e1dffcaacd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:24:56.683768 13070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 688ef6aca9dd43888d908e1dffcaacd9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:56.685114 13070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 688ef6aca9dd43888d908e1dffcaacd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688ef6aca9dd43888d908e1dffcaacd9" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 40833 } }
Aug 10 07:24:56 dist-test-slave-ffrd krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:24:56.685539 13070 leader_election.cc:302] T 00000000000000000000000000000000 P 688ef6aca9dd43888d908e1dffcaacd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 688ef6aca9dd43888d908e1dffcaacd9; no voters:
I20260810 07:24:56.685906 13070 leader_election.cc:288] T 00000000000000000000000000000000 P 688ef6aca9dd43888d908e1dffcaacd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:24:56.686002 13079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 688ef6aca9dd43888d908e1dffcaacd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:56.686210 13079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 688ef6aca9dd43888d908e1dffcaacd9 [term 1 LEADER]: Becoming Leader. State: Replica: 688ef6aca9dd43888d908e1dffcaacd9, State: Running, Role: LEADER
I20260810 07:24:56.686659 13079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 688ef6aca9dd43888d908e1dffcaacd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688ef6aca9dd43888d908e1dffcaacd9" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 40833 } }
I20260810 07:24:56.687418 13070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 688ef6aca9dd43888d908e1dffcaacd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:24:56.688858 13082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 688ef6aca9dd43888d908e1dffcaacd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "688ef6aca9dd43888d908e1dffcaacd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688ef6aca9dd43888d908e1dffcaacd9" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 40833 } } }
I20260810 07:24:56.689054 13082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 688ef6aca9dd43888d908e1dffcaacd9 [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:56.688858 13079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 688ef6aca9dd43888d908e1dffcaacd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 688ef6aca9dd43888d908e1dffcaacd9. Latest consensus state: current_term: 1 leader_uuid: "688ef6aca9dd43888d908e1dffcaacd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "688ef6aca9dd43888d908e1dffcaacd9" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 40833 } } }
I20260810 07:24:56.689908 13079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 688ef6aca9dd43888d908e1dffcaacd9 [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:56.693063 13074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.667589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39576 (local address 127.12.157.254:40833)
0810 07:24:56.668129 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:56.668149 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:56.669150 (+ 1001us) server_negotiation.cc:407] Connection header received
0810 07:24:56.673798 (+ 4648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:56.673814 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:56.674196 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:56.674499 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:56.677521 (+ 3022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:56.678568 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:56.680316 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:56.680897 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:56.681122 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:56.686413 (+ 5291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:56.686420 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:56.686435 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:56.686469 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:56.688770 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:56.689646 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:56.689655 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:56.689661 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:56.689818 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:56.690366 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:56.690370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:56.690372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:56.690920 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:56.691137 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:56.691739 (+ 602us) server_negotiation.cc:299] Negotiation successful
0810 07:24:56.692049 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":150,"threads_started":1}
W20260810 07:24:56.693531 13106 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 688ef6aca9dd43888d908e1dffcaacd9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:24:56.693604 13106 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:24:56.693713 13093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:24:56.694644 13093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:24:56.700825 13093 catalog_manager.cc:1347] Generated new cluster ID: 5853bc4d5f4a4611acdd64d8d38e6454
I20260810 07:24:56.700919 13093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:24:56.730322 13093 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:24:56.731652 13093 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:24:56.740684 13093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 688ef6aca9dd43888d908e1dffcaacd9: Generated new TSK 0
I20260810 07:24:56.741478 13093 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
I20260810 07:24:56.777211 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:0
--local_ip_for_outbound_sockets=127.12.157.193
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:40833
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:42335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:56.903889 13115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:56.904152 13115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:56.904198 13115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:56.908325 13115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:56.908402 13115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:56.908495 13115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:24:56.913933 13115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:42335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:40833
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:24:56.915089 13115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:56.916137 13115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:56.926287 13127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:56.926286 13125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:56.926286 13122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:56.926693 13115 server_base.cc:1034] running on GCE node
I20260810 07:24:56.927189 13115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:56.927841 13115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:56.929031 13115 hybrid_clock.cc:648] HybridClock initialized: now 1786346696929006 us; error 26 us; skew 500 ppm
Aug 10 07:24:56 dist-test-slave-ffrd krb5kdc[12933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346696, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:56.931845 13115 init.cc:377] Logged in from keytab as kudu/127.12.157.193@KRBTEST.COM (short username kudu)
I20260810 07:24:56.932806 13115 webserver.cc:460] Webserver started at http://127.12.157.193:39677/ using document root <none> and password file <none>
I20260810 07:24:56.933454 13115 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:56.933552 13115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:56.933790 13115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:56.936091 13115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-0/data/instance:
uuid: "2ebef937a2ed45d8b058bebbf9d81b14"
format_stamp: "Formatted at 2026-08-10 07:24:56 on dist-test-slave-ffrd"
server_key: "1a8d1c74870e2c12e131ddc981dea82e"
server_key_iv: "3336639eafda902096f1ba779870fc56"
server_key_version: "encryptionkey@0"
I20260810 07:24:56.936815 13115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance:
uuid: "2ebef937a2ed45d8b058bebbf9d81b14"
format_stamp: "Formatted at 2026-08-10 07:24:56 on dist-test-slave-ffrd"
server_key: "1a8d1c74870e2c12e131ddc981dea82e"
server_key_iv: "3336639eafda902096f1ba779870fc56"
server_key_version: "encryptionkey@0"
I20260810 07:24:56.940760 13115 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 07:24:56.950938 13137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:56.956523 13115 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.006s sys 0.000s
I20260810 07:24:56.956727 13115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "2ebef937a2ed45d8b058bebbf9d81b14"
format_stamp: "Formatted at 2026-08-10 07:24:56 on dist-test-slave-ffrd"
server_key: "1a8d1c74870e2c12e131ddc981dea82e"
server_key_iv: "3336639eafda902096f1ba779870fc56"
server_key_version: "encryptionkey@0"
I20260810 07:24:56.956892 13115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:56.977978 13115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:56.981417 13115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:56.981666 13115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:56.982342 13115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:56.983246 13115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:56.983325 13115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:56.983400 13115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:56.983448 13115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:56.993564 13115 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:42585
I20260810 07:24:56.993597 13318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:42585 every 8 connection(s)
I20260810 07:24:56.994489 13115 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:24:56.996398 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 13115
I20260810 07:24:56.996515 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance
I20260810 07:24:56.996778 12919 external_mini_cluster.cc:1428] Setting key 30a7365ead240638cb1bf7e3abf48204
Aug 10 07:24:57 dist-test-slave-ffrd krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346696, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.157.194@KRBTEST.COM; defaulting to no policy
I20260810 07:24:57.011098 13074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.996540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:33581 (local address 127.12.157.254:40833)
0810 07:24:56.996683 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:56.996688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:56.998282 (+ 1594us) server_negotiation.cc:407] Connection header received
0810 07:24:56.999302 (+ 1020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:56.999308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:56.999371 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:56.999489 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:57.000909 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:57.001610 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:57.003455 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:57.003708 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:57.003798 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:57.006950 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:57.006953 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:57.006956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:57.006982 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:57.008664 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:57.009298 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:57.009302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:57.009304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:57.009429 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:57.009844 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:57.009846 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:57.009848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:57.010053 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:57.010143 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:57.010795 (+ 652us) server_negotiation.cc:299] Negotiation successful
0810 07:24:57.010938 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.194@KRBTEST.COM" created.
I20260810 07:24:57.012044 13322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:56.997010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:40833 (local address 127.12.157.193:33581)
0810 07:24:56.998096 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:56.998154 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:56.999080 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:56.999690 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:56.999697 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:57.000085 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:57.000686 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:57.000710 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:57.001756 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:57.001763 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:57.003289 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:57.003305 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:57.003897 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:57.003904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:57.004152 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:57.004873 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:57.004903 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:57.006754 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:57.008794 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:57.008801 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:57.008818 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:57.009188 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:57.009577 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:57.009580 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:57.009582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:57.009737 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:57.010168 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:57.010174 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:57.010483 (+ 309us) client_negotiation.cc:769] Sending connection context
0810 07:24:57.010666 (+ 183us) client_negotiation.cc:240] Negotiation successful
0810 07:24:57.010964 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"thread_start_us":657,"threads_started":1}
I20260810 07:24:57.013710 13319 heartbeater.cc:347] Connected to a master server at 127.12.157.254:40833
I20260810 07:24:57.013998 13319 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:57.014544 13319 heartbeater.cc:511] Master 127.12.157.254:40833 requested a full tablet report, sending...
I20260810 07:24:57.016148 13010 ts_manager.cc:194] Registered new tserver with Master: 2ebef937a2ed45d8b058bebbf9d81b14 (127.12.157.193:42585)
I20260810 07:24:57.017380 13010 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.193@KRBTEST.COM'} at 127.12.157.193:33581
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
I20260810 07:24:57.028560 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:0
--local_ip_for_outbound_sockets=127.12.157.194
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:40833
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:42335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:57.169659 13325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:57.169953 13325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:57.170001 13325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:57.174489 13325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:57.174575 13325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:57.174693 13325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:24:57.180923 13325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:42335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:40833
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:24:57.182130 13325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:57.183226 13325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:57.194476 13333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:57.194537 13336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:57.194480 13332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:57.194893 13325 server_base.cc:1034] running on GCE node
I20260810 07:24:57.195348 13325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:57.196059 13325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:57.197227 13325 hybrid_clock.cc:648] HybridClock initialized: now 1786346697197149 us; error 88 us; skew 500 ppm
Aug 10 07:24:57 dist-test-slave-ffrd krb5kdc[12933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346697, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:57.200191 13325 init.cc:377] Logged in from keytab as kudu/127.12.157.194@KRBTEST.COM (short username kudu)
I20260810 07:24:57.201180 13325 webserver.cc:460] Webserver started at http://127.12.157.194:44947/ using document root <none> and password file <none>
I20260810 07:24:57.201732 13325 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:57.201802 13325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:57.201977 13325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:57.204114 13325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-1/data/instance:
uuid: "ce9e3db8932044898bb82e25343aed97"
format_stamp: "Formatted at 2026-08-10 07:24:57 on dist-test-slave-ffrd"
server_key: "b4e7030cab03bd5f029a304fc4131bd9"
server_key_iv: "1950ed77f19c4753d19ce58e56e223f0"
server_key_version: "encryptionkey@0"
I20260810 07:24:57.204653 13325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance:
uuid: "ce9e3db8932044898bb82e25343aed97"
format_stamp: "Formatted at 2026-08-10 07:24:57 on dist-test-slave-ffrd"
server_key: "b4e7030cab03bd5f029a304fc4131bd9"
server_key_iv: "1950ed77f19c4753d19ce58e56e223f0"
server_key_version: "encryptionkey@0"
I20260810 07:24:57.209446 13325 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260810 07:24:57.212249 13348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:57.214128 13325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:24:57.214331 13325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "ce9e3db8932044898bb82e25343aed97"
format_stamp: "Formatted at 2026-08-10 07:24:57 on dist-test-slave-ffrd"
server_key: "b4e7030cab03bd5f029a304fc4131bd9"
server_key_iv: "1950ed77f19c4753d19ce58e56e223f0"
server_key_version: "encryptionkey@0"
I20260810 07:24:57.214478 13325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:57.235272 13325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:57.239041 13325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:57.239316 13325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:57.240139 13325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:57.241313 13325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:57.241387 13325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:57.241461 13325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:57.241515 13325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:57.261873 13325 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:38559
I20260810 07:24:57.262044 13530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:38559 every 8 connection(s)
I20260810 07:24:57.262897 13325 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:24:57.267290 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 13325
I20260810 07:24:57.267457 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance
I20260810 07:24:57.267735 12919 external_mini_cluster.cc:1428] Setting key 9ecd29268129977528b01a65ee3931f3
Aug 10 07:24:57 dist-test-slave-ffrd krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346697, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:24:57.281275 13074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:57.265203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:35989 (local address 127.12.157.254:40833)
0810 07:24:57.265474 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:57.265478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:57.266559 (+ 1081us) server_negotiation.cc:407] Connection header received
0810 07:24:57.268552 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:57.268556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:57.268611 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:57.268709 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:57.270558 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:57.271386 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:57.272850 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:57.273301 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:57.273443 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:57.277034 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:57.277037 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:57.277039 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:57.277064 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:57.278644 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:57.279381 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:57.279383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:57.279385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:57.279459 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:57.279932 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:57.279935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:57.279936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:57.280110 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:57.280185 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:57.281037 (+ 852us) server_negotiation.cc:299] Negotiation successful
0810 07:24:57.281146 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
WARNING: no policy specified for kudu/127.12.157.195@KRBTEST.COM; defaulting to no policy
I20260810 07:24:57.282687 13534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:57.265614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:40833 (local address 127.12.157.194:35989)
0810 07:24:57.266293 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:57.266337 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:57.268345 (+ 2008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:57.268910 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:57.268918 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:57.269698 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:57.270379 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:57.270396 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:57.271544 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:57.271547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:57.272684 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:57.272695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:57.273494 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:57.273497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:57.273839 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:57.274622 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:57.274649 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:57.276788 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:57.278776 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:57.278795 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:57.278831 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:57.279255 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:57.279560 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:57.279566 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:57.279569 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:57.279818 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:57.280240 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:57.280249 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:57.280652 (+ 403us) client_negotiation.cc:769] Sending connection context
0810 07:24:57.280928 (+ 276us) client_negotiation.cc:240] Negotiation successful
0810 07:24:57.281438 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":233,"threads_started":1}
I20260810 07:24:57.284060 13531 heartbeater.cc:347] Connected to a master server at 127.12.157.254:40833
I20260810 07:24:57.284349 13531 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.195@KRBTEST.COM" created.
I20260810 07:24:57.284916 13531 heartbeater.cc:511] Master 127.12.157.254:40833 requested a full tablet report, sending...
I20260810 07:24:57.286219 13010 ts_manager.cc:194] Registered new tserver with Master: ce9e3db8932044898bb82e25343aed97 (127.12.157.194:38559)
I20260810 07:24:57.287197 13010 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.194@KRBTEST.COM'} at 127.12.157.194:35989
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
I20260810 07:24:57.304589 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:0
--local_ip_for_outbound_sockets=127.12.157.195
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:40833
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:42335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:57.473431 13540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:57.473716 13540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:57.473763 13540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:57.478263 13540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:57.478354 13540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:57.478439 13540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:24:57.486639 13540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:42335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:40833
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:24:57.488314 13540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:57.489876 13540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:57.504136 13555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:57.504187 13554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:57.504513 13557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:57.504828 13540 server_base.cc:1034] running on GCE node
I20260810 07:24:57.505290 13540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:57.506116 13540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:57.507337 13540 hybrid_clock.cc:648] HybridClock initialized: now 1786346697507313 us; error 44 us; skew 500 ppm
Aug 10 07:24:57 dist-test-slave-ffrd krb5kdc[12933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346697, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:57.511513 13540 init.cc:377] Logged in from keytab as kudu/127.12.157.195@KRBTEST.COM (short username kudu)
I20260810 07:24:57.512470 13540 webserver.cc:460] Webserver started at http://127.12.157.195:42681/ using document root <none> and password file <none>
I20260810 07:24:57.513159 13540 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:57.513224 13540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:57.513451 13540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:57.515821 13540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-2/data/instance:
uuid: "f43b19db6bdd4d3199fc474ff9f5e07d"
format_stamp: "Formatted at 2026-08-10 07:24:57 on dist-test-slave-ffrd"
server_key: "ce86691cbe943f4af0aca4df289507d9"
server_key_iv: "bb4a17df8ac28d9a9147ef511e629a22"
server_key_version: "encryptionkey@0"
I20260810 07:24:57.516481 13540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance:
uuid: "f43b19db6bdd4d3199fc474ff9f5e07d"
format_stamp: "Formatted at 2026-08-10 07:24:57 on dist-test-slave-ffrd"
server_key: "ce86691cbe943f4af0aca4df289507d9"
server_key_iv: "bb4a17df8ac28d9a9147ef511e629a22"
server_key_version: "encryptionkey@0"
I20260810 07:24:57.520913 13540 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 07:24:57.523797 13570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:57.525205 13540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:24:57.525341 13540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "f43b19db6bdd4d3199fc474ff9f5e07d"
format_stamp: "Formatted at 2026-08-10 07:24:57 on dist-test-slave-ffrd"
server_key: "ce86691cbe943f4af0aca4df289507d9"
server_key_iv: "bb4a17df8ac28d9a9147ef511e629a22"
server_key_version: "encryptionkey@0"
I20260810 07:24:57.525466 13540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:57.539592 13540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:57.543497 13540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:57.543808 13540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:57.544653 13540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:57.545950 13540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:57.546128 13540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:57.546211 13540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:57.546265 13540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:57.564412 13540 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:43815
I20260810 07:24:57.564499 13767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:43815 every 8 connection(s)
I20260810 07:24:57.565486 13540 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:24:57.567142 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 13540
I20260810 07:24:57.567279 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance
I20260810 07:24:57.567638 12919 external_mini_cluster.cc:1428] Setting key e4ac433694be1560da868ef502bf2df3
Aug 10 07:24:57 dist-test-slave-ffrd krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346697, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:24:57.591948 13074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:57.573066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:37159 (local address 127.12.157.254:40833)
0810 07:24:57.573209 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:57.573214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:57.577521 (+ 4307us) server_negotiation.cc:407] Connection header received
0810 07:24:57.579825 (+ 2304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:57.579829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:57.579882 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:57.579972 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:57.581976 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:57.582672 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:57.584635 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:57.584850 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:57.585011 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:57.588143 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:57.588146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:57.588148 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:57.588174 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:57.589799 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:57.590344 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:57.590347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:57.590349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:57.590437 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:57.590797 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:57.590800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:57.590802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:57.591008 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:57.591116 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:57.591649 (+ 533us) server_negotiation.cc:299] Negotiation successful
0810 07:24:57.591770 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 07:24:57.592808 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:57.568284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:40833 (local address 127.12.157.195:37159)
0810 07:24:57.577266 (+ 8982us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:57.577369 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:57.579508 (+ 2139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:57.580247 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:57.580261 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:57.580978 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:57.581796 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:57.581818 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:57.582826 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:57.582831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:57.584492 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:57.584506 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:57.585063 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:57.585069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:57.585326 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:57.586034 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:57.586061 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:57.587965 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:57.589938 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:57.589945 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:57.589961 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:57.590232 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:57.590539 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:57.590542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:57.590543 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:57.590691 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:57.591128 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:57.591137 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:57.591397 (+ 260us) client_negotiation.cc:769] Sending connection context
0810 07:24:57.591589 (+ 192us) client_negotiation.cc:240] Negotiation successful
0810 07:24:57.591931 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8590,"thread_start_us":174,"threads_started":1}
I20260810 07:24:57.594007 13768 heartbeater.cc:347] Connected to a master server at 127.12.157.254:40833
I20260810 07:24:57.594269 13768 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:57.594811 13768 heartbeater.cc:511] Master 127.12.157.254:40833 requested a full tablet report, sending...
I20260810 07:24:57.595839 13010 ts_manager.cc:194] Registered new tserver with Master: f43b19db6bdd4d3199fc474ff9f5e07d (127.12.157.195:43815)
I20260810 07:24:57.596413 13010 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.195@KRBTEST.COM'} at 127.12.157.195:37159
I20260810 07:24:57.606829 12919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 07:24:57 dist-test-slave-ffrd krb5kdc[12933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346697, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 07:24:57 dist-test-slave-ffrd krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346697, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.157.193@KRBTEST.COM
I20260810 07:24:57.636672 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:57.626484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55530 (local address 127.12.157.193:42585)
0810 07:24:57.626967 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:57.626975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:57.627008 (+ 33us) server_negotiation.cc:407] Connection header received
0810 07:24:57.627058 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:57.627063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:57.627253 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:57.627448 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:57.628140 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:57.628924 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:57.629817 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:57.630259 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:57.630385 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:57.633035 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:57.633044 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:57.633058 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:57.633083 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:57.634687 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:57.635176 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:57.635190 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:57.635191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:57.635284 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:57.635575 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:57.635579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:57.635580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:57.635839 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:57.636063 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:57.636335 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 07:24:57.636500 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":102,"threads_started":1}
W20260810 07:24:57.637883 13189 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55530
Aug 10 07:24:57 dist-test-slave-ffrd krb5kdc[12933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 07:24:57 dist-test-slave-ffrd krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.193@KRBTEST.COM
I20260810 07:24:57.666934 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:57.657261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55532 (local address 127.12.157.193:42585)
0810 07:24:57.657411 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:57.657415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:57.657575 (+ 160us) server_negotiation.cc:407] Connection header received
0810 07:24:57.657691 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:57.657694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:57.657739 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:57.657817 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:57.658532 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:57.659052 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:57.660177 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:57.660427 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:57.660524 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:57.663411 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:57.663414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:57.663416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:57.663443 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:57.665133 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:57.665587 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:57.665590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:57.665591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:57.665671 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:57.665961 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:57.665964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:57.665965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:57.666133 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:57.666240 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:57.666526 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 07:24:57.666649 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 07:24:57.668118 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 13115
I20260810 07:24:57.679716 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 13325
I20260810 07:24:57.701694 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 13540
I20260810 07:24:57.718969 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 12956
2026-08-10T07:24:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2738 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:24:57 dist-test-slave-ffrd krb5kdc[13786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:24:57 dist-test-slave-ffrd krb5kdc[13786](info): set up 2 sockets
Aug 10 07:24:57 dist-test-slave-ffrd krb5kdc[13786](info): commencing operation
krb5kdc: starting...
W20260810 07:24:59.000154 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.255s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:24:59 dist-test-slave-ffrd krb5kdc[13786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:24:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:24:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:24:59.135560 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:36475
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:40019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:36475
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:59.253321 13817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:59.253682 13817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:59.253773 13817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:59.258471 13817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:24:59.258566 13817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:59.258599 13817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:59.258633 13817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:24:59.258678 13817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:24:59.264593 13817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:40019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:36475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:36475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:24:59.265823 13817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:59.266945 13817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:59.276098 13833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:59.276098 13829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:59.276362 13817 server_base.cc:1034] running on GCE node
W20260810 07:24:59.276103 13828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:59.277235 13817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:59.278502 13817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:59.279712 13817 hybrid_clock.cc:648] HybridClock initialized: now 1786346699279690 us; error 35 us; skew 500 ppm
Aug 10 07:24:59 dist-test-slave-ffrd krb5kdc[13786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346699, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:59.282712 13817 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:24:59.283686 13817 webserver.cc:460] Webserver started at http://127.12.157.254:32837/ using document root <none> and password file <none>
I20260810 07:24:59.284284 13817 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:59.284349 13817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:59.284577 13817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:59.286767 13817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "0643a50bfc7d4a22a1401aeb6672f43d"
format_stamp: "Formatted at 2026-08-10 07:24:59 on dist-test-slave-ffrd"
server_key: "5e1758a54b55c50e488261c9d1477b20"
server_key_iv: "50d3166aa77fe69e70a05236e789ba1c"
server_key_version: "encryptionkey@0"
I20260810 07:24:59.287384 13817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "0643a50bfc7d4a22a1401aeb6672f43d"
format_stamp: "Formatted at 2026-08-10 07:24:59 on dist-test-slave-ffrd"
server_key: "5e1758a54b55c50e488261c9d1477b20"
server_key_iv: "50d3166aa77fe69e70a05236e789ba1c"
server_key_version: "encryptionkey@0"
I20260810 07:24:59.291291 13817 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 07:24:59.294226 13844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:59.295559 13817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:24:59.295696 13817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "0643a50bfc7d4a22a1401aeb6672f43d"
format_stamp: "Formatted at 2026-08-10 07:24:59 on dist-test-slave-ffrd"
server_key: "5e1758a54b55c50e488261c9d1477b20"
server_key_iv: "50d3166aa77fe69e70a05236e789ba1c"
server_key_version: "encryptionkey@0"
I20260810 07:24:59.295822 13817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:59.307951 13817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:59.311463 13817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:59.311684 13817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:59.319126 13817 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:36475
I20260810 07:24:59.319180 13933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:36475 every 8 connection(s)
I20260810 07:24:59.320104 13817 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:24:59.323040 13936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:24:59.323223 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 13817
I20260810 07:24:59.323351 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:24:59.323632 12919 external_mini_cluster.cc:1428] Setting key 743d728f617fef2462a84be3fb6d510a
I20260810 07:24:59.330479 13936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0643a50bfc7d4a22a1401aeb6672f43d: Bootstrap starting.
I20260810 07:24:59.333039 13936 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0643a50bfc7d4a22a1401aeb6672f43d: Neither blocks nor log segments found. Creating new log.
Aug 10 07:24:59 dist-test-slave-ffrd krb5kdc[13786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:24:59.333992 13936 log.cc:824] T 00000000000000000000000000000000 P 0643a50bfc7d4a22a1401aeb6672f43d: Log is configured to *not* fsync() on all Append() calls
I20260810 07:24:59.335953 13936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0643a50bfc7d4a22a1401aeb6672f43d: No bootstrap required, opened a new log
I20260810 07:24:59.339115 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:59.325123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49354 (local address 127.12.157.254:36475)
0810 07:24:59.325576 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:59.325596 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:59.325638 (+ 42us) server_negotiation.cc:407] Connection header received
0810 07:24:59.326596 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:59.326621 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:59.327056 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:59.327423 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:59.328159 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.329685 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.330602 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.331264 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.331433 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:59.334267 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:59.334272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:59.334287 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:59.334316 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:59.336267 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:59.336749 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:59.336755 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:59.336760 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:59.336877 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:59.337205 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:59.337213 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:59.337214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:59.337609 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:59.337812 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:59.338103 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 07:24:59.338397 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":129,"threads_started":1}
I20260810 07:24:59.339767 13936 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0643a50bfc7d4a22a1401aeb6672f43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0643a50bfc7d4a22a1401aeb6672f43d" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 36475 } }
I20260810 07:24:59.339982 13936 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0643a50bfc7d4a22a1401aeb6672f43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:24:59.340051 13936 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0643a50bfc7d4a22a1401aeb6672f43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0643a50bfc7d4a22a1401aeb6672f43d, State: Initialized, Role: FOLLOWER
I20260810 07:24:59.340477 13936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0643a50bfc7d4a22a1401aeb6672f43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0643a50bfc7d4a22a1401aeb6672f43d" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 36475 } }
I20260810 07:24:59.340600 13936 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0643a50bfc7d4a22a1401aeb6672f43d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:24:59.340688 13936 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0643a50bfc7d4a22a1401aeb6672f43d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:24:59.340780 13936 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0643a50bfc7d4a22a1401aeb6672f43d [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:24:59.341953 13936 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0643a50bfc7d4a22a1401aeb6672f43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0643a50bfc7d4a22a1401aeb6672f43d" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 36475 } }
I20260810 07:24:59.342301 13936 leader_election.cc:302] T 00000000000000000000000000000000 P 0643a50bfc7d4a22a1401aeb6672f43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0643a50bfc7d4a22a1401aeb6672f43d; no voters:
I20260810 07:24:59.342622 13936 leader_election.cc:288] T 00000000000000000000000000000000 P 0643a50bfc7d4a22a1401aeb6672f43d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:24:59.342792 13943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0643a50bfc7d4a22a1401aeb6672f43d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:24:59.343102 13943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0643a50bfc7d4a22a1401aeb6672f43d [term 1 LEADER]: Becoming Leader. State: Replica: 0643a50bfc7d4a22a1401aeb6672f43d, State: Running, Role: LEADER
I20260810 07:24:59.343415 13943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0643a50bfc7d4a22a1401aeb6672f43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0643a50bfc7d4a22a1401aeb6672f43d" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 36475 } }
I20260810 07:24:59.343916 13936 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0643a50bfc7d4a22a1401aeb6672f43d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:24:59.345101 13945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0643a50bfc7d4a22a1401aeb6672f43d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0643a50bfc7d4a22a1401aeb6672f43d. Latest consensus state: current_term: 1 leader_uuid: "0643a50bfc7d4a22a1401aeb6672f43d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0643a50bfc7d4a22a1401aeb6672f43d" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 36475 } } }
I20260810 07:24:59.345216 13945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0643a50bfc7d4a22a1401aeb6672f43d [sys.catalog]: This master's current role is: LEADER
I20260810 07:24:59.345750 13944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0643a50bfc7d4a22a1401aeb6672f43d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0643a50bfc7d4a22a1401aeb6672f43d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0643a50bfc7d4a22a1401aeb6672f43d" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 36475 } } }
I20260810 07:24:59.345860 13944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0643a50bfc7d4a22a1401aeb6672f43d [sys.catalog]: This master's current role is: LEADER
W20260810 07:24:59.348692 13970 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0643a50bfc7d4a22a1401aeb6672f43d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:24:59.348758 13970 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:24:59.348845 13960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:24:59.349576 13960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:24:59.356866 13960 catalog_manager.cc:1347] Generated new cluster ID: fa0464778f614c2fb57f05aeb06403b6
I20260810 07:24:59.356969 13960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:24:59.367832 13960 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:24:59.368769 13960 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:24:59.375916 13960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0643a50bfc7d4a22a1401aeb6672f43d: Generated new TSK 0
I20260810 07:24:59.376588 13960 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
I20260810 07:24:59.409905 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:0
--local_ip_for_outbound_sockets=127.12.157.193
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:36475
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:40019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:59.529950 13977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:59.530225 13977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:59.530273 13977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:59.534487 13977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:59.534559 13977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:59.534622 13977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:24:59.540254 13977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:40019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:36475
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:24:59.541533 13977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:59.542637 13977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:59.553472 13984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:59.553653 13983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:59.553758 13977 server_base.cc:1034] running on GCE node
W20260810 07:24:59.553472 13988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:59.554356 13977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:59.555074 13977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:59.556262 13977 hybrid_clock.cc:648] HybridClock initialized: now 1786346699556234 us; error 47 us; skew 500 ppm
Aug 10 07:24:59 dist-test-slave-ffrd krb5kdc[13786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346699, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:59.559629 13977 init.cc:377] Logged in from keytab as kudu/127.12.157.193@KRBTEST.COM (short username kudu)
I20260810 07:24:59.560606 13977 webserver.cc:460] Webserver started at http://127.12.157.193:41949/ using document root <none> and password file <none>
I20260810 07:24:59.561290 13977 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:59.561400 13977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:59.561655 13977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:59.564134 13977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-0/data/instance:
uuid: "8213a5839982478fa63dfc747d4f47a2"
format_stamp: "Formatted at 2026-08-10 07:24:59 on dist-test-slave-ffrd"
server_key: "67087be5c4183b84a1f6af7ee1912311"
server_key_iv: "d02090dd030956006827e1295df5bc79"
server_key_version: "encryptionkey@0"
I20260810 07:24:59.564762 13977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance:
uuid: "8213a5839982478fa63dfc747d4f47a2"
format_stamp: "Formatted at 2026-08-10 07:24:59 on dist-test-slave-ffrd"
server_key: "67087be5c4183b84a1f6af7ee1912311"
server_key_iv: "d02090dd030956006827e1295df5bc79"
server_key_version: "encryptionkey@0"
I20260810 07:24:59.569029 13977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 07:24:59.571722 14002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:59.573320 13977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 07:24:59.573477 13977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "8213a5839982478fa63dfc747d4f47a2"
format_stamp: "Formatted at 2026-08-10 07:24:59 on dist-test-slave-ffrd"
server_key: "67087be5c4183b84a1f6af7ee1912311"
server_key_iv: "d02090dd030956006827e1295df5bc79"
server_key_version: "encryptionkey@0"
I20260810 07:24:59.573601 13977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:59.603471 13977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:59.606710 13977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:59.606943 13977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:59.607656 13977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:59.608527 13977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:59.608604 13977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:59.608680 13977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:59.608734 13977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:59.618158 13977 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:32931
I20260810 07:24:59.618191 14188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:32931 every 8 connection(s)
I20260810 07:24:59.619078 13977 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:24:59.627974 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 13977
I20260810 07:24:59.628104 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance
I20260810 07:24:59.628408 12919 external_mini_cluster.cc:1428] Setting key 4d2251cfee3211ae8bdc8554cbbb093b
Aug 10 07:24:59 dist-test-slave-ffrd krb5kdc[13786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346699, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:24:59.634047 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:59.621209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:58979 (local address 127.12.157.254:36475)
0810 07:24:59.621382 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:59.621388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:59.622245 (+ 857us) server_negotiation.cc:407] Connection header received
0810 07:24:59.623545 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:59.623548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:59.623600 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:59.623687 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:59.624921 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.625531 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.626693 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.626961 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.627045 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:59.630321 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:59.630325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:59.630327 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:59.630350 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:59.631843 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:59.632379 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:59.632384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:59.632385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:59.632471 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:59.632860 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:59.632864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:59.632866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:59.633115 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:59.633223 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:59.633755 (+ 532us) server_negotiation.cc:299] Negotiation successful
0810 07:24:59.633885 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 07:24:59.634938 14191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:59.621583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:36475 (local address 127.12.157.193:58979)
0810 07:24:59.622073 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:59.622122 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:59.623307 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:59.623867 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:59.623874 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:59.624248 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:59.624752 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:59.624765 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.625659 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.625662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:59.626574 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:59.626584 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.627103 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.627106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:59.627251 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:59.627983 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:59.628008 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:59.630128 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:59.631972 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:59.631978 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:59.631994 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:59.632266 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:59.632567 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:59.632571 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:59.632573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:59.632724 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:59.633233 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:59.633242 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:59.633511 (+ 269us) client_negotiation.cc:769] Sending connection context
0810 07:24:59.633712 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 07:24:59.634067 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":134,"threads_started":1}
I20260810 07:24:59.636278 14189 heartbeater.cc:347] Connected to a master server at 127.12.157.254:36475
I20260810 07:24:59.636548 14189 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:59.637115 14189 heartbeater.cc:511] Master 127.12.157.254:36475 requested a full tablet report, sending...
I20260810 07:24:59.638863 13876 ts_manager.cc:194] Registered new tserver with Master: 8213a5839982478fa63dfc747d4f47a2 (127.12.157.193:32931)
WARNING: no policy specified for kudu/127.12.157.194@KRBTEST.COM; defaulting to no policy
I20260810 07:24:59.640221 13876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.193@KRBTEST.COM'} at 127.12.157.193:58979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
I20260810 07:24:59.659996 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:0
--local_ip_for_outbound_sockets=127.12.157.194
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:36475
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:40019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:24:59.778112 14196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:24:59.778412 14196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:24:59.778489 14196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:24:59.782516 14196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:24:59.782619 14196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:24:59.782714 14196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:24:59.787981 14196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:40019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:36475
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:24:59.789144 14196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:24:59.790217 14196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:24:59.800413 14206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:59.800446 14211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:24:59.800443 14205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:24:59.801052 14196 server_base.cc:1034] running on GCE node
I20260810 07:24:59.801443 14196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:24:59.802122 14196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:24:59.803335 14196 hybrid_clock.cc:648] HybridClock initialized: now 1786346699803325 us; error 32 us; skew 500 ppm
Aug 10 07:24:59 dist-test-slave-ffrd krb5kdc[13786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346699, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:24:59.806396 14196 init.cc:377] Logged in from keytab as kudu/127.12.157.194@KRBTEST.COM (short username kudu)
I20260810 07:24:59.807376 14196 webserver.cc:460] Webserver started at http://127.12.157.194:40169/ using document root <none> and password file <none>
I20260810 07:24:59.807956 14196 fs_manager.cc:359] Metadata directory not provided
I20260810 07:24:59.808050 14196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:24:59.808303 14196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:24:59.810500 14196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-1/data/instance:
uuid: "84ed2c6240334c4ca69641819fb255c1"
format_stamp: "Formatted at 2026-08-10 07:24:59 on dist-test-slave-ffrd"
server_key: "b96b8dd3e48b123df8cbff92091a1ff8"
server_key_iv: "0440eedd7590b8d71429002b153ad1da"
server_key_version: "encryptionkey@0"
I20260810 07:24:59.811053 14196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance:
uuid: "84ed2c6240334c4ca69641819fb255c1"
format_stamp: "Formatted at 2026-08-10 07:24:59 on dist-test-slave-ffrd"
server_key: "b96b8dd3e48b123df8cbff92091a1ff8"
server_key_iv: "0440eedd7590b8d71429002b153ad1da"
server_key_version: "encryptionkey@0"
I20260810 07:24:59.814882 14196 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 07:24:59.817417 14223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:59.818666 14196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:24:59.818816 14196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "84ed2c6240334c4ca69641819fb255c1"
format_stamp: "Formatted at 2026-08-10 07:24:59 on dist-test-slave-ffrd"
server_key: "b96b8dd3e48b123df8cbff92091a1ff8"
server_key_iv: "0440eedd7590b8d71429002b153ad1da"
server_key_version: "encryptionkey@0"
I20260810 07:24:59.818933 14196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:24:59.840655 14196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:24:59.843963 14196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:24:59.844245 14196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:24:59.845055 14196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:24:59.846093 14196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:24:59.846167 14196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:59.846237 14196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:24:59.846292 14196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:24:59.855656 14196 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:34633
I20260810 07:24:59.855691 14410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:34633 every 8 connection(s)
I20260810 07:24:59.856607 14196 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:24:59.858233 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 14196
I20260810 07:24:59.858366 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance
I20260810 07:24:59.858652 12919 external_mini_cluster.cc:1428] Setting key 9341a7f9cea13817d2e1d5b8233035d2
Aug 10 07:24:59 dist-test-slave-ffrd krb5kdc[13786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346699, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.195@KRBTEST.COM" created.
I20260810 07:24:59.873891 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:59.858579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:48667 (local address 127.12.157.254:36475)
0810 07:24:59.858725 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 07:24:59.858729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:24:59.859678 (+ 949us) server_negotiation.cc:407] Connection header received
0810 07:24:59.861238 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:24:59.861242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:24:59.861295 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:24:59.861373 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:24:59.862786 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.863377 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.865220 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.865673 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.865787 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:59.869402 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:24:59.869405 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:24:59.869407 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:24:59.869434 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:24:59.871235 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:59.871878 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:59.871882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:59.871883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:59.871971 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:24:59.872349 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:24:59.872513 (+ 164us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:24:59.872517 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:24:59.872839 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:24:59.872985 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 07:24:59.873641 (+ 656us) server_negotiation.cc:299] Negotiation successful
0810 07:24:59.873761 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 07:24:59.875156 14414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:24:59.859066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:36475 (local address 127.12.157.194:48667)
0810 07:24:59.859544 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0810 07:24:59.859586 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 07:24:59.861033 (+ 1447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:24:59.861542 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:24:59.861548 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:24:59.861924 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:24:59.862579 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:59.862601 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.863522 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.863525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:59.865065 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:24:59.865081 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:24:59.865868 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:24:59.865873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:24:59.866096 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:24:59.866861 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:24:59.866890 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:24:59.869135 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:24:59.871392 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:59.871401 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:59.871420 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:59.871722 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:59.872063 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:24:59.872066 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:24:59.872067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:24:59.872207 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:24:59.873021 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:24:59.873027 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:24:59.873351 (+ 324us) client_negotiation.cc:769] Sending connection context
0810 07:24:59.873562 (+ 211us) client_negotiation.cc:240] Negotiation successful
0810 07:24:59.873981 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":118,"threads_started":1}
I20260810 07:24:59.876675 14411 heartbeater.cc:347] Connected to a master server at 127.12.157.254:36475
I20260810 07:24:59.877051 14411 heartbeater.cc:464] Registering TS with master...
I20260810 07:24:59.877591 14411 heartbeater.cc:511] Master 127.12.157.254:36475 requested a full tablet report, sending...
I20260810 07:24:59.878722 13876 ts_manager.cc:194] Registered new tserver with Master: 84ed2c6240334c4ca69641819fb255c1 (127.12.157.194:34633)
I20260810 07:24:59.879316 13876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.194@KRBTEST.COM'} at 127.12.157.194:48667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
I20260810 07:24:59.892647 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:0
--local_ip_for_outbound_sockets=127.12.157.195
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:36475
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:40019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:00.005256 14420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:00.005564 14420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:00.005613 14420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:00.009799 14420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:00.009866 14420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:00.009930 14420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:25:00.016148 14420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:40019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:36475
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:00.017375 14420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:00.018445 14420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:00.028776 14434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:00.028971 14420 server_base.cc:1034] running on GCE node
W20260810 07:25:00.029202 14439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:00.029206 14428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:00.029757 14420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:00.030408 14420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:00.031613 14420 hybrid_clock.cc:648] HybridClock initialized: now 1786346700031597 us; error 30 us; skew 500 ppm
Aug 10 07:25:00 dist-test-slave-ffrd krb5kdc[13786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346700, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:00.034754 14420 init.cc:377] Logged in from keytab as kudu/127.12.157.195@KRBTEST.COM (short username kudu)
I20260810 07:25:00.035694 14420 webserver.cc:460] Webserver started at http://127.12.157.195:39697/ using document root <none> and password file <none>
I20260810 07:25:00.036305 14420 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:00.036404 14420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:00.036648 14420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:00.038801 14420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-2/data/instance:
uuid: "b3276153c895417ca0bc6cd2c30a14e4"
format_stamp: "Formatted at 2026-08-10 07:25:00 on dist-test-slave-ffrd"
server_key: "4a873575847cc86f5e66aeeda119a10f"
server_key_iv: "510868136a91d3cb5a6a01ee9be2f7e6"
server_key_version: "encryptionkey@0"
I20260810 07:25:00.039407 14420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance:
uuid: "b3276153c895417ca0bc6cd2c30a14e4"
format_stamp: "Formatted at 2026-08-10 07:25:00 on dist-test-slave-ffrd"
server_key: "4a873575847cc86f5e66aeeda119a10f"
server_key_iv: "510868136a91d3cb5a6a01ee9be2f7e6"
server_key_version: "encryptionkey@0"
I20260810 07:25:00.043598 14420 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 07:25:00.046486 14449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:00.047780 14420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 07:25:00.047946 14420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "b3276153c895417ca0bc6cd2c30a14e4"
format_stamp: "Formatted at 2026-08-10 07:25:00 on dist-test-slave-ffrd"
server_key: "4a873575847cc86f5e66aeeda119a10f"
server_key_iv: "510868136a91d3cb5a6a01ee9be2f7e6"
server_key_version: "encryptionkey@0"
I20260810 07:25:00.048070 14420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:00.070463 14420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:00.073791 14420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:00.074003 14420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:00.074631 14420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:00.075496 14420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:00.075572 14420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:00.075645 14420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:00.075699 14420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:00.086321 14420 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:41861
I20260810 07:25:00.086347 14635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:41861 every 8 connection(s)
I20260810 07:25:00.087306 14420 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:25:00.090041 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 14420
I20260810 07:25:00.090191 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance
I20260810 07:25:00.090529 12919 external_mini_cluster.cc:1428] Setting key 60ad1f5fae56e245744c84c78b338b25
Aug 10 07:25:00 dist-test-slave-ffrd krb5kdc[13786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346700, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:00.103318 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:00.089536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:35705 (local address 127.12.157.254:36475)
0810 07:25:00.089715 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:00.089719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:00.090751 (+ 1032us) server_negotiation.cc:407] Connection header received
0810 07:25:00.092197 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:00.092200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:00.092261 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:00.092349 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:00.093887 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.094450 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.095842 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.096070 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.096163 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:00.099089 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:00.099092 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:00.099094 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:00.099121 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:00.101004 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:00.101593 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:00.101596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:00.101598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:00.101687 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:00.102072 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:00.102074 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:00.102076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:00.102311 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:00.102490 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:00.103074 (+ 584us) server_negotiation.cc:299] Negotiation successful
0810 07:25:00.103188 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 07:25:00.104394 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:00.089910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:36475 (local address 127.12.157.195:35705)
0810 07:25:00.090555 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:00.090619 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:00.092001 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:00.092561 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:00.092570 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:00.093141 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:00.093715 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:00.093730 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.094624 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.094627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:00.095691 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:00.095702 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.096218 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.096221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:00.096388 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:00.097135 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:00.097154 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:00.098878 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:00.101163 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:00.101170 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:00.101185 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:00.101472 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:00.101788 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:00.101791 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:00.101792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:00.101972 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:00.102550 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:00.102557 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:00.102831 (+ 274us) client_negotiation.cc:769] Sending connection context
0810 07:25:00.103007 (+ 176us) client_negotiation.cc:240] Negotiation successful
0810 07:25:00.103297 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":165,"threads_started":1}
I20260810 07:25:00.105983 14637 heartbeater.cc:347] Connected to a master server at 127.12.157.254:36475
I20260810 07:25:00.106212 14637 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:00.106750 14637 heartbeater.cc:511] Master 127.12.157.254:36475 requested a full tablet report, sending...
I20260810 07:25:00.107887 13875 ts_manager.cc:194] Registered new tserver with Master: b3276153c895417ca0bc6cd2c30a14e4 (127.12.157.195:41861)
I20260810 07:25:00.108632 13875 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.195@KRBTEST.COM'} at 127.12.157.195:35705
I20260810 07:25:00.118153 12919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:25:00.130731 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:00.122130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49358 (local address 127.12.157.254:36475)
0810 07:25:00.122278 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:00.122288 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:00.122512 (+ 224us) server_negotiation.cc:407] Connection header received
0810 07:25:00.122581 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:00.122583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:00.122631 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:00.122711 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:00.123619 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.124196 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.125196 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.125436 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.125533 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:00.127022 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:00.127026 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:00.127028 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:00.127055 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:00.128682 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:00.129256 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:00.129260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:00.129261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:00.129349 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:00.129754 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:00.129757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:00.129758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:00.130036 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:00.130158 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:00.130482 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 07:25:00.130607 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 07:25:00.137427 13875 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:25:00.140637 13875 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:25:00.156865 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:00.150489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:32931 (local address 127.0.0.1:56548)
0810 07:25:00.151127 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:00.151183 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:00.151476 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:00.152358 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:00.152364 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:00.152415 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:00.152559 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:00.152567 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.153819 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.153823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:00.154958 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:00.154970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.156084 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.156087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:00.156201 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:00.156240 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 07:25:00.156362 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 07:25:00.156597 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":123,"threads_started":1}
I20260810 07:25:00.156875 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:00.150670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56548 (local address 127.12.157.193:32931)
0810 07:25:00.151177 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:00.151185 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:00.151373 (+ 188us) server_negotiation.cc:407] Connection header received
0810 07:25:00.151748 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:00.151755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:00.152008 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:00.152165 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:00.152709 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.153683 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.155106 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.155953 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.156098 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:00.156255 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:00.156392 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 07:25:00.156597 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":69,"threads_started":1}
I20260810 07:25:00.157596 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:00.150331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:41861 (local address 127.0.0.1:55762)
0810 07:25:00.150926 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:00.151223 (+ 297us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:00.151476 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:00.152811 (+ 1335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:00.152816 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:00.152848 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:00.152971 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:00.152978 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.154487 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.154491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:00.155620 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:00.155631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.157142 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.157146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:00.157247 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:00.157271 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 07:25:00.157361 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 07:25:00.157436 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":5,"spinlock_wait_cycles":308608,"thread_start_us":135,"threads_started":1}
I20260810 07:25:00.158054 14654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:00.151175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:34633 (local address 127.0.0.1:34032)
0810 07:25:00.151493 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:00.151509 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:00.151593 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:00.152592 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:00.152595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:00.152614 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:00.152777 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:00.152788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.155051 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.155054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:00.156437 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:00.156448 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.157648 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.157652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:00.157773 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:00.157797 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 07:25:00.157867 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 07:25:00.157938 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":119,"threads_started":1}
I20260810 07:25:00.158128 14655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:00.151624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34032 (local address 127.12.157.194:34633)
0810 07:25:00.152109 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:00.152123 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:00.152145 (+ 22us) server_negotiation.cc:407] Connection header received
0810 07:25:00.152223 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:00.152227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:00.152378 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:00.152600 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:00.152923 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.154621 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.156598 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.157500 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.157671 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:00.157835 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:00.157893 (+ 58us) server_negotiation.cc:299] Negotiation successful
0810 07:25:00.157984 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":138,"threads_started":1}
I20260810 07:25:00.158476 14657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:00.151678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55762 (local address 127.12.157.195:41861)
0810 07:25:00.152255 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:00.152267 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:00.152290 (+ 23us) server_negotiation.cc:407] Connection header received
0810 07:25:00.152364 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:00.152369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:00.152559 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:00.152705 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:00.153107 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.154327 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.155934 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.156793 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.156921 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:00.157076 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:00.157897 (+ 821us) server_negotiation.cc:299] Negotiation successful
0810 07:25:00.157995 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":147,"threads_started":1}
I20260810 07:25:00.160892 14528 tablet_service.cc:1467] Processing CreateTablet for tablet 16becc956c614d7fa8bad62cf481101f (DEFAULT_TABLE table=test-table [id=a84d56301e0e42e98bd699ba57a09acc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:00.160739 14301 tablet_service.cc:1467] Processing CreateTablet for tablet 16becc956c614d7fa8bad62cf481101f (DEFAULT_TABLE table=test-table [id=a84d56301e0e42e98bd699ba57a09acc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:00.160741 14078 tablet_service.cc:1467] Processing CreateTablet for tablet 16becc956c614d7fa8bad62cf481101f (DEFAULT_TABLE table=test-table [id=a84d56301e0e42e98bd699ba57a09acc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:00.161940 14301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16becc956c614d7fa8bad62cf481101f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:00.161949 14078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16becc956c614d7fa8bad62cf481101f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:00.161954 14528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16becc956c614d7fa8bad62cf481101f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:00.169795 14658 tablet_bootstrap.cc:492] T 16becc956c614d7fa8bad62cf481101f P 84ed2c6240334c4ca69641819fb255c1: Bootstrap starting.
I20260810 07:25:00.171197 14660 tablet_bootstrap.cc:492] T 16becc956c614d7fa8bad62cf481101f P 8213a5839982478fa63dfc747d4f47a2: Bootstrap starting.
I20260810 07:25:00.172922 14658 tablet_bootstrap.cc:654] T 16becc956c614d7fa8bad62cf481101f P 84ed2c6240334c4ca69641819fb255c1: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:00.173362 14660 tablet_bootstrap.cc:654] T 16becc956c614d7fa8bad62cf481101f P 8213a5839982478fa63dfc747d4f47a2: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:00.173367 14662 tablet_bootstrap.cc:492] T 16becc956c614d7fa8bad62cf481101f P b3276153c895417ca0bc6cd2c30a14e4: Bootstrap starting.
I20260810 07:25:00.174124 14658 log.cc:824] T 16becc956c614d7fa8bad62cf481101f P 84ed2c6240334c4ca69641819fb255c1: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:00.174582 14660 log.cc:824] T 16becc956c614d7fa8bad62cf481101f P 8213a5839982478fa63dfc747d4f47a2: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:00.175890 14662 tablet_bootstrap.cc:654] T 16becc956c614d7fa8bad62cf481101f P b3276153c895417ca0bc6cd2c30a14e4: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:00.176186 14658 tablet_bootstrap.cc:492] T 16becc956c614d7fa8bad62cf481101f P 84ed2c6240334c4ca69641819fb255c1: No bootstrap required, opened a new log
I20260810 07:25:00.176417 14658 ts_tablet_manager.cc:1397] T 16becc956c614d7fa8bad62cf481101f P 84ed2c6240334c4ca69641819fb255c1: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260810 07:25:00.177677 14662 log.cc:824] T 16becc956c614d7fa8bad62cf481101f P b3276153c895417ca0bc6cd2c30a14e4: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:00.180565 14662 tablet_bootstrap.cc:492] T 16becc956c614d7fa8bad62cf481101f P b3276153c895417ca0bc6cd2c30a14e4: No bootstrap required, opened a new log
I20260810 07:25:00.180785 14662 ts_tablet_manager.cc:1397] T 16becc956c614d7fa8bad62cf481101f P b3276153c895417ca0bc6cd2c30a14e4: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260810 07:25:00.180895 14658 raft_consensus.cc:348] T 16becc956c614d7fa8bad62cf481101f P 84ed2c6240334c4ca69641819fb255c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3276153c895417ca0bc6cd2c30a14e4" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41861 } } peers { permanent_uuid: "8213a5839982478fa63dfc747d4f47a2" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 32931 } } peers { permanent_uuid: "84ed2c6240334c4ca69641819fb255c1" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 34633 } }
I20260810 07:25:00.181221 14658 raft_consensus.cc:374] T 16becc956c614d7fa8bad62cf481101f P 84ed2c6240334c4ca69641819fb255c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:00.181272 14658 raft_consensus.cc:729] T 16becc956c614d7fa8bad62cf481101f P 84ed2c6240334c4ca69641819fb255c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84ed2c6240334c4ca69641819fb255c1, State: Initialized, Role: FOLLOWER
I20260810 07:25:00.181795 14658 consensus_queue.cc:260] T 16becc956c614d7fa8bad62cf481101f P 84ed2c6240334c4ca69641819fb255c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3276153c895417ca0bc6cd2c30a14e4" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41861 } } peers { permanent_uuid: "8213a5839982478fa63dfc747d4f47a2" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 32931 } } peers { permanent_uuid: "84ed2c6240334c4ca69641819fb255c1" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 34633 } }
I20260810 07:25:00.182443 14660 tablet_bootstrap.cc:492] T 16becc956c614d7fa8bad62cf481101f P 8213a5839982478fa63dfc747d4f47a2: No bootstrap required, opened a new log
I20260810 07:25:00.182618 14411 heartbeater.cc:503] Master 127.12.157.254:36475 was elected leader, sending a full tablet report...
I20260810 07:25:00.182657 14660 ts_tablet_manager.cc:1397] T 16becc956c614d7fa8bad62cf481101f P 8213a5839982478fa63dfc747d4f47a2: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260810 07:25:00.182931 14658 ts_tablet_manager.cc:1428] T 16becc956c614d7fa8bad62cf481101f P 84ed2c6240334c4ca69641819fb255c1: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 07:25:00.186169 14662 raft_consensus.cc:348] T 16becc956c614d7fa8bad62cf481101f P b3276153c895417ca0bc6cd2c30a14e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3276153c895417ca0bc6cd2c30a14e4" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41861 } } peers { permanent_uuid: "8213a5839982478fa63dfc747d4f47a2" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 32931 } } peers { permanent_uuid: "84ed2c6240334c4ca69641819fb255c1" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 34633 } }
I20260810 07:25:00.186509 14662 raft_consensus.cc:374] T 16becc956c614d7fa8bad62cf481101f P b3276153c895417ca0bc6cd2c30a14e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:00.186583 14662 raft_consensus.cc:729] T 16becc956c614d7fa8bad62cf481101f P b3276153c895417ca0bc6cd2c30a14e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3276153c895417ca0bc6cd2c30a14e4, State: Initialized, Role: FOLLOWER
I20260810 07:25:00.187096 14662 consensus_queue.cc:260] T 16becc956c614d7fa8bad62cf481101f P b3276153c895417ca0bc6cd2c30a14e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3276153c895417ca0bc6cd2c30a14e4" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41861 } } peers { permanent_uuid: "8213a5839982478fa63dfc747d4f47a2" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 32931 } } peers { permanent_uuid: "84ed2c6240334c4ca69641819fb255c1" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 34633 } }
I20260810 07:25:00.187557 14660 raft_consensus.cc:348] T 16becc956c614d7fa8bad62cf481101f P 8213a5839982478fa63dfc747d4f47a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3276153c895417ca0bc6cd2c30a14e4" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41861 } } peers { permanent_uuid: "8213a5839982478fa63dfc747d4f47a2" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 32931 } } peers { permanent_uuid: "84ed2c6240334c4ca69641819fb255c1" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 34633 } }
I20260810 07:25:00.187805 14660 raft_consensus.cc:374] T 16becc956c614d7fa8bad62cf481101f P 8213a5839982478fa63dfc747d4f47a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:00.187863 14660 raft_consensus.cc:729] T 16becc956c614d7fa8bad62cf481101f P 8213a5839982478fa63dfc747d4f47a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8213a5839982478fa63dfc747d4f47a2, State: Initialized, Role: FOLLOWER
I20260810 07:25:00.187861 14637 heartbeater.cc:503] Master 127.12.157.254:36475 was elected leader, sending a full tablet report...
I20260810 07:25:00.188192 14662 ts_tablet_manager.cc:1428] T 16becc956c614d7fa8bad62cf481101f P b3276153c895417ca0bc6cd2c30a14e4: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 07:25:00.188268 14660 consensus_queue.cc:260] T 16becc956c614d7fa8bad62cf481101f P 8213a5839982478fa63dfc747d4f47a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3276153c895417ca0bc6cd2c30a14e4" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41861 } } peers { permanent_uuid: "8213a5839982478fa63dfc747d4f47a2" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 32931 } } peers { permanent_uuid: "84ed2c6240334c4ca69641819fb255c1" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 34633 } }
I20260810 07:25:00.189011 14189 heartbeater.cc:503] Master 127.12.157.254:36475 was elected leader, sending a full tablet report...
I20260810 07:25:00.189222 14660 ts_tablet_manager.cc:1428] T 16becc956c614d7fa8bad62cf481101f P 8213a5839982478fa63dfc747d4f47a2: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260810 07:25:00.338413 14638 tablet.cc:2369] T 16becc956c614d7fa8bad62cf481101f P b3276153c895417ca0bc6cd2c30a14e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:25:00.354871 14674 raft_consensus.cc:482] T 16becc956c614d7fa8bad62cf481101f P b3276153c895417ca0bc6cd2c30a14e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:25:00.355078 14674 raft_consensus.cc:504] T 16becc956c614d7fa8bad62cf481101f P b3276153c895417ca0bc6cd2c30a14e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3276153c895417ca0bc6cd2c30a14e4" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41861 } } peers { permanent_uuid: "8213a5839982478fa63dfc747d4f47a2" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 32931 } } peers { permanent_uuid: "84ed2c6240334c4ca69641819fb255c1" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 34633 } }
I20260810 07:25:00.356279 14674 leader_election.cc:288] T 16becc956c614d7fa8bad62cf481101f P b3276153c895417ca0bc6cd2c30a14e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8213a5839982478fa63dfc747d4f47a2 (127.12.157.193:32931), 84ed2c6240334c4ca69641819fb255c1 (127.12.157.194:34633)
W20260810 07:25:00.357872 14412 tablet.cc:2369] T 16becc956c614d7fa8bad62cf481101f P 84ed2c6240334c4ca69641819fb255c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:25:00.361061 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:00.356604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:32931 (local address 127.12.157.195:39367)
0810 07:25:00.356793 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:00.356813 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:00.356921 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:00.357280 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:00.357284 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:00.357308 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:00.357490 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:00.357499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.358444 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.358450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:00.359784 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:00.359796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.360523 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.360527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:00.360641 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:00.360708 (+ 67us) client_negotiation.cc:769] Sending connection context
0810 07:25:00.360799 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 07:25:00.360888 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":1}
I20260810 07:25:00.361092 14655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:00.356837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:35089 (local address 127.12.157.194:34633)
0810 07:25:00.356995 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:00.357000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:00.357227 (+ 227us) server_negotiation.cc:407] Connection header received
0810 07:25:00.357314 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:00.357316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:00.357368 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:00.357474 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:00.357757 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.358513 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.359970 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.360397 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.360540 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:00.360596 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:00.360883 (+ 287us) server_negotiation.cc:299] Negotiation successful
0810 07:25:00.360983 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 07:25:00.361086 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:00.356682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:39367 (local address 127.12.157.193:32931)
0810 07:25:00.356835 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:00.356839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:00.356869 (+ 30us) server_negotiation.cc:407] Connection header received
0810 07:25:00.357089 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:00.357092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:00.357161 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:00.357259 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:00.357626 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.358320 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.359932 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.360386 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.360523 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:00.360585 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:00.360843 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 07:25:00.360933 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 07:25:00.361099 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:00.356646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:34633 (local address 127.12.157.195:35089)
0810 07:25:00.357092 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:00.357106 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:00.357226 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:00.357481 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:00.357483 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:00.357498 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:00.357637 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:00.357644 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.358624 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.358627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:00.359810 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:00.359816 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.360547 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.360550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:00.360732 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:00.360759 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 07:25:00.360862 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 07:25:00.360933 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":79,"thread_start_us":157,"threads_started":1}
I20260810 07:25:00.361959 14117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16becc956c614d7fa8bad62cf481101f" candidate_uuid: "b3276153c895417ca0bc6cd2c30a14e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8213a5839982478fa63dfc747d4f47a2" is_pre_election: true
I20260810 07:25:00.361956 14341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16becc956c614d7fa8bad62cf481101f" candidate_uuid: "b3276153c895417ca0bc6cd2c30a14e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84ed2c6240334c4ca69641819fb255c1" is_pre_election: true
I20260810 07:25:00.362289 14117 raft_consensus.cc:2454] T 16becc956c614d7fa8bad62cf481101f P 8213a5839982478fa63dfc747d4f47a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3276153c895417ca0bc6cd2c30a14e4 in term 0.
I20260810 07:25:00.362288 14341 raft_consensus.cc:2454] T 16becc956c614d7fa8bad62cf481101f P 84ed2c6240334c4ca69641819fb255c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3276153c895417ca0bc6cd2c30a14e4 in term 0.
I20260810 07:25:00.362797 14453 leader_election.cc:302] T 16becc956c614d7fa8bad62cf481101f P b3276153c895417ca0bc6cd2c30a14e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8213a5839982478fa63dfc747d4f47a2, b3276153c895417ca0bc6cd2c30a14e4; no voters:
I20260810 07:25:00.363041 14674 raft_consensus.cc:2781] T 16becc956c614d7fa8bad62cf481101f P b3276153c895417ca0bc6cd2c30a14e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:25:00.363145 14674 raft_consensus.cc:482] T 16becc956c614d7fa8bad62cf481101f P b3276153c895417ca0bc6cd2c30a14e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:25:00.363206 14674 raft_consensus.cc:3037] T 16becc956c614d7fa8bad62cf481101f P b3276153c895417ca0bc6cd2c30a14e4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:00.364264 14674 raft_consensus.cc:504] T 16becc956c614d7fa8bad62cf481101f P b3276153c895417ca0bc6cd2c30a14e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3276153c895417ca0bc6cd2c30a14e4" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41861 } } peers { permanent_uuid: "8213a5839982478fa63dfc747d4f47a2" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 32931 } } peers { permanent_uuid: "84ed2c6240334c4ca69641819fb255c1" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 34633 } }
I20260810 07:25:00.364641 14674 leader_election.cc:288] T 16becc956c614d7fa8bad62cf481101f P b3276153c895417ca0bc6cd2c30a14e4 [CANDIDATE]: Term 1 election: Requested vote from peers 8213a5839982478fa63dfc747d4f47a2 (127.12.157.193:32931), 84ed2c6240334c4ca69641819fb255c1 (127.12.157.194:34633)
I20260810 07:25:00.365039 14341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16becc956c614d7fa8bad62cf481101f" candidate_uuid: "b3276153c895417ca0bc6cd2c30a14e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84ed2c6240334c4ca69641819fb255c1"
I20260810 07:25:00.365077 14117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16becc956c614d7fa8bad62cf481101f" candidate_uuid: "b3276153c895417ca0bc6cd2c30a14e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8213a5839982478fa63dfc747d4f47a2"
I20260810 07:25:00.365162 14341 raft_consensus.cc:3037] T 16becc956c614d7fa8bad62cf481101f P 84ed2c6240334c4ca69641819fb255c1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:00.365219 14117 raft_consensus.cc:3037] T 16becc956c614d7fa8bad62cf481101f P 8213a5839982478fa63dfc747d4f47a2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:00.366283 14117 raft_consensus.cc:2454] T 16becc956c614d7fa8bad62cf481101f P 8213a5839982478fa63dfc747d4f47a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3276153c895417ca0bc6cd2c30a14e4 in term 1.
I20260810 07:25:00.366451 14341 raft_consensus.cc:2454] T 16becc956c614d7fa8bad62cf481101f P 84ed2c6240334c4ca69641819fb255c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3276153c895417ca0bc6cd2c30a14e4 in term 1.
I20260810 07:25:00.366652 14453 leader_election.cc:302] T 16becc956c614d7fa8bad62cf481101f P b3276153c895417ca0bc6cd2c30a14e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8213a5839982478fa63dfc747d4f47a2, b3276153c895417ca0bc6cd2c30a14e4; no voters:
I20260810 07:25:00.366842 14674 raft_consensus.cc:2781] T 16becc956c614d7fa8bad62cf481101f P b3276153c895417ca0bc6cd2c30a14e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:00.367053 14674 raft_consensus.cc:686] T 16becc956c614d7fa8bad62cf481101f P b3276153c895417ca0bc6cd2c30a14e4 [term 1 LEADER]: Becoming Leader. State: Replica: b3276153c895417ca0bc6cd2c30a14e4, State: Running, Role: LEADER
I20260810 07:25:00.367403 14674 consensus_queue.cc:237] T 16becc956c614d7fa8bad62cf481101f P b3276153c895417ca0bc6cd2c30a14e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3276153c895417ca0bc6cd2c30a14e4" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41861 } } peers { permanent_uuid: "8213a5839982478fa63dfc747d4f47a2" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 32931 } } peers { permanent_uuid: "84ed2c6240334c4ca69641819fb255c1" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 34633 } }
W20260810 07:25:00.370399 14190 tablet.cc:2369] T 16becc956c614d7fa8bad62cf481101f P 8213a5839982478fa63dfc747d4f47a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:25:00.370764 13875 catalog_manager.cc:5495] T 16becc956c614d7fa8bad62cf481101f P b3276153c895417ca0bc6cd2c30a14e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3276153c895417ca0bc6cd2c30a14e4 (127.12.157.195). New cstate: current_term: 1 leader_uuid: "b3276153c895417ca0bc6cd2c30a14e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3276153c895417ca0bc6cd2c30a14e4" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8213a5839982478fa63dfc747d4f47a2" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 32931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84ed2c6240334c4ca69641819fb255c1" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 34633 } health_report { overall_health: UNKNOWN } } }
Aug 10 07:25:00 dist-test-slave-ffrd krb5kdc[13786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.193@KRBTEST.COM
I20260810 07:25:00.396590 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:00.387799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56562 (local address 127.12.157.193:32931)
0810 07:25:00.387973 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:00.387977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:00.388059 (+ 82us) server_negotiation.cc:407] Connection header received
0810 07:25:00.388169 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:00.388171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:00.388214 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:00.388293 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:00.388997 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.389513 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.390354 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.390555 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.390630 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:00.393125 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:00.393136 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:00.393152 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:00.393177 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:00.394863 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:00.395303 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:00.395308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:00.395310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:00.395415 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:00.395673 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:00.395675 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:00.395676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:00.395917 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:00.396077 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:00.396242 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 07:25:00.396352 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
Aug 10 07:25:00 dist-test-slave-ffrd krb5kdc[13786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346700, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 07:25:00 dist-test-slave-ffrd krb5kdc[13786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346700, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.157.193@KRBTEST.COM
I20260810 07:25:00.426520 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:00.417189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56578 (local address 127.12.157.193:32931)
0810 07:25:00.417358 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:00.417363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:00.417560 (+ 197us) server_negotiation.cc:407] Connection header received
0810 07:25:00.417668 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:00.417671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:00.417722 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:00.417820 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:00.418677 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.419324 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.420223 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.420456 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.420571 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:00.423256 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:00.423259 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:00.423261 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:00.423297 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:00.424710 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:00.425234 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:00.425238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:00.425239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:00.425352 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:00.425671 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:00.425673 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:00.425674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:00.425851 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:00.425964 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:00.426240 (+ 276us) server_negotiation.cc:299] Negotiation successful
0810 07:25:00.426346 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 07:25:00 dist-test-slave-ffrd krb5kdc[13786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 07:25:00 dist-test-slave-ffrd krb5kdc[13786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.193@KRBTEST.COM
I20260810 07:25:00.455510 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:00.446593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56586 (local address 127.12.157.193:32931)
0810 07:25:00.446768 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:00.446773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:00.446894 (+ 121us) server_negotiation.cc:407] Connection header received
0810 07:25:00.447068 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:00.447072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:00.447126 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:00.447216 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:00.447927 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.448445 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.449344 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:00.449583 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:00.449698 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:00.452239 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:00.452242 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:00.452244 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:00.452271 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:00.453806 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:00.454204 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:00.454206 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:00.454207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:00.454285 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:00.454537 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:00.454539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:00.454540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:00.454704 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:00.454807 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:00.455229 (+ 422us) server_negotiation.cc:299] Negotiation successful
0810 07:25:00.455387 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260810 07:25:00.456902 14048 tablet_service.cc:2908] Rejecting scan request for tablet 16becc956c614d7fa8bad62cf481101f: Uninitialized: clean time has not yet been initialized
I20260810 07:25:00.459322 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 13977
I20260810 07:25:00.473716 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 14196
I20260810 07:25:00.490680 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 14420
I20260810 07:25:00.509079 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 13817
2026-08-10T07:25:00Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2791 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:25:00 dist-test-slave-ffrd krb5kdc[14704](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:25:00 dist-test-slave-ffrd krb5kdc[14704](info): set up 2 sockets
Aug 10 07:25:00 dist-test-slave-ffrd krb5kdc[14704](info): commencing operation
krb5kdc: starting...
W20260810 07:25:01.791172 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.254s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:25:01 dist-test-slave-ffrd krb5kdc[14704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:25:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:25:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:25:01.940623 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:34629
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:46815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:34629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:02.055074 14722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:02.055382 14722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:02.055472 14722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:02.059532 14722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:25:02.059628 14722 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 07:25:02.059662 14722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:02.059705 14722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:02.059746 14722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:25:02.059796 14722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:25:02.065184 14722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:46815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:34629
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:34629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:02.066340 14722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:02.067480 14722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:02.076749 14737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:02.076962 14722 server_base.cc:1034] running on GCE node
W20260810 07:25:02.076752 14730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:02.077158 14732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:02.077888 14722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:02.079097 14722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:02.080286 14722 hybrid_clock.cc:648] HybridClock initialized: now 1786346702080258 us; error 45 us; skew 500 ppm
Aug 10 07:25:02 dist-test-slave-ffrd krb5kdc[14704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346702, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:02.083385 14722 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:25:02.084316 14722 webserver.cc:460] Webserver started at http://127.12.157.254:35035/ using document root <none> and password file <none>
I20260810 07:25:02.084901 14722 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:02.085029 14722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:02.085233 14722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:02.087409 14722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "69c4220dc13d4817a4497084935f8707"
format_stamp: "Formatted at 2026-08-10 07:25:02 on dist-test-slave-ffrd"
server_key: "0c514c176de4925f057fb962961e79b3"
server_key_iv: "a21d5e31736856453332f7c1ee6528e5"
server_key_version: "encryptionkey@0"
I20260810 07:25:02.088013 14722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "69c4220dc13d4817a4497084935f8707"
format_stamp: "Formatted at 2026-08-10 07:25:02 on dist-test-slave-ffrd"
server_key: "0c514c176de4925f057fb962961e79b3"
server_key_iv: "a21d5e31736856453332f7c1ee6528e5"
server_key_version: "encryptionkey@0"
I20260810 07:25:02.091986 14722 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260810 07:25:02.094628 14747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:02.095942 14722 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 07:25:02.096047 14722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "69c4220dc13d4817a4497084935f8707"
format_stamp: "Formatted at 2026-08-10 07:25:02 on dist-test-slave-ffrd"
server_key: "0c514c176de4925f057fb962961e79b3"
server_key_iv: "a21d5e31736856453332f7c1ee6528e5"
server_key_version: "encryptionkey@0"
I20260810 07:25:02.096130 14722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:02.106537 14722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:02.109867 14722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:02.110028 14722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:02.118609 14722 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:34629
I20260810 07:25:02.118597 14839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:34629 every 8 connection(s)
I20260810 07:25:02.119624 14722 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:25:02.122678 14843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:02.128618 14843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c4220dc13d4817a4497084935f8707: Bootstrap starting.
I20260810 07:25:02.129011 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 14722
I20260810 07:25:02.129151 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:25:02.129459 12919 external_mini_cluster.cc:1428] Setting key 267b663d47ceb8752f559348bc345399
I20260810 07:25:02.131538 14843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69c4220dc13d4817a4497084935f8707: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:02.132350 14843 log.cc:824] T 00000000000000000000000000000000 P 69c4220dc13d4817a4497084935f8707: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:02.135078 14843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c4220dc13d4817a4497084935f8707: No bootstrap required, opened a new log
I20260810 07:25:02.139072 14843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69c4220dc13d4817a4497084935f8707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c4220dc13d4817a4497084935f8707" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 34629 } }
I20260810 07:25:02.139379 14843 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69c4220dc13d4817a4497084935f8707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Aug 10 07:25:02 dist-test-slave-ffrd krb5kdc[14704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:02.139472 14843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69c4220dc13d4817a4497084935f8707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69c4220dc13d4817a4497084935f8707, State: Initialized, Role: FOLLOWER
I20260810 07:25:02.140107 14843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69c4220dc13d4817a4497084935f8707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c4220dc13d4817a4497084935f8707" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 34629 } }
I20260810 07:25:02.140288 14843 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69c4220dc13d4817a4497084935f8707 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:25:02.140385 14843 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69c4220dc13d4817a4497084935f8707 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:25:02.140525 14843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69c4220dc13d4817a4497084935f8707 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:02.141858 14843 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69c4220dc13d4817a4497084935f8707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c4220dc13d4817a4497084935f8707" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 34629 } }
I20260810 07:25:02.142253 14843 leader_election.cc:302] T 00000000000000000000000000000000 P 69c4220dc13d4817a4497084935f8707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69c4220dc13d4817a4497084935f8707; no voters:
I20260810 07:25:02.142676 14843 leader_election.cc:288] T 00000000000000000000000000000000 P 69c4220dc13d4817a4497084935f8707 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:25:02.142802 14851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69c4220dc13d4817a4497084935f8707 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:02.143090 14851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69c4220dc13d4817a4497084935f8707 [term 1 LEADER]: Becoming Leader. State: Replica: 69c4220dc13d4817a4497084935f8707, State: Running, Role: LEADER
I20260810 07:25:02.143401 14851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69c4220dc13d4817a4497084935f8707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c4220dc13d4817a4497084935f8707" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 34629 } }
I20260810 07:25:02.144299 14843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69c4220dc13d4817a4497084935f8707 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:25:02.145469 14852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c4220dc13d4817a4497084935f8707 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69c4220dc13d4817a4497084935f8707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c4220dc13d4817a4497084935f8707" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 34629 } } }
I20260810 07:25:02.145612 14852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c4220dc13d4817a4497084935f8707 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:02.145898 14853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c4220dc13d4817a4497084935f8707 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69c4220dc13d4817a4497084935f8707. Latest consensus state: current_term: 1 leader_uuid: "69c4220dc13d4817a4497084935f8707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c4220dc13d4817a4497084935f8707" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 34629 } } }
I20260810 07:25:02.146016 14853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c4220dc13d4817a4497084935f8707 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:02.146117 14868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:25:02.146749 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:02.130923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48302 (local address 127.12.157.254:34629)
0810 07:25:02.131386 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:02.131404 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:02.131442 (+ 38us) server_negotiation.cc:407] Connection header received
0810 07:25:02.132175 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:02.132194 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:02.132596 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:02.133018 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:02.134079 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.135494 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.136470 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.137061 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.137263 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:02.140201 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:02.140206 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:02.140221 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:02.140305 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:02.142504 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:02.143102 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:02.143110 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:02.143118 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:02.143270 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:02.143790 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:02.143794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:02.143796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:02.144387 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:02.144604 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:02.145018 (+ 414us) server_negotiation.cc:299] Negotiation successful
0810 07:25:02.145524 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":145,"threads_started":1}
I20260810 07:25:02.149842 14868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:25:02.156095 14868 catalog_manager.cc:1347] Generated new cluster ID: 1f28042167fb4f559def0c8b30ac8232
I20260810 07:25:02.156181 14868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:25:02.169477 14868 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:25:02.170768 14868 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:25:02.180601 14868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69c4220dc13d4817a4497084935f8707: Generated new TSK 0
I20260810 07:25:02.181522 14868 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
I20260810 07:25:02.218215 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:0
--local_ip_for_outbound_sockets=127.12.157.193
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:34629
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:46815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:02.342448 14887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:02.342818 14887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:02.342918 14887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:02.347445 14887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:02.347543 14887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:02.347617 14887 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 07:25:02.347702 14887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:25:02.353418 14887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:46815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.157.254:34629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:02.354924 14887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:02.356076 14887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:02.366276 14900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:02.366294 14895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:02.366560 14897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:02.367100 14887 server_base.cc:1034] running on GCE node
I20260810 07:25:02.367607 14887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:02.368366 14887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:02.369545 14887 hybrid_clock.cc:648] HybridClock initialized: now 1786346702369492 us; error 66 us; skew 500 ppm
Aug 10 07:25:02 dist-test-slave-ffrd krb5kdc[14704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346702, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:02.372570 14887 init.cc:377] Logged in from keytab as kudu/127.12.157.193@KRBTEST.COM (short username kudu)
I20260810 07:25:02.373569 14887 webserver.cc:460] Webserver started at http://127.12.157.193:45289/ using document root <none> and password file <none>
I20260810 07:25:02.374120 14887 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:02.374181 14887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:02.374351 14887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:02.376438 14887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-0/data/instance:
uuid: "e48974bc7ca24cf2b14a5b40c3861044"
format_stamp: "Formatted at 2026-08-10 07:25:02 on dist-test-slave-ffrd"
server_key: "6b22ced1fed459fb9c41cb6dec8318e1"
server_key_iv: "eca51b0470a31ff2c2fb0e0620b0b358"
server_key_version: "encryptionkey@0"
I20260810 07:25:02.377051 14887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance:
uuid: "e48974bc7ca24cf2b14a5b40c3861044"
format_stamp: "Formatted at 2026-08-10 07:25:02 on dist-test-slave-ffrd"
server_key: "6b22ced1fed459fb9c41cb6dec8318e1"
server_key_iv: "eca51b0470a31ff2c2fb0e0620b0b358"
server_key_version: "encryptionkey@0"
I20260810 07:25:02.380815 14887 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 07:25:02.383389 14910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:02.384696 14887 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 07:25:02.384847 14887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "e48974bc7ca24cf2b14a5b40c3861044"
format_stamp: "Formatted at 2026-08-10 07:25:02 on dist-test-slave-ffrd"
server_key: "6b22ced1fed459fb9c41cb6dec8318e1"
server_key_iv: "eca51b0470a31ff2c2fb0e0620b0b358"
server_key_version: "encryptionkey@0"
I20260810 07:25:02.385041 14887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:02.396293 14887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:02.399700 14887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:02.399926 14887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:02.401638 14887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:02.401690 14887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:02.401731 14887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:02.401747 14887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 07:25:02 dist-test-slave-ffrd krb5kdc[14704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346702, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:02.413254 14887 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:46371
I20260810 07:25:02.413393 15114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:46371 every 8 connection(s)
I20260810 07:25:02.414306 14887 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:25:02.416177 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:02.402424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:47999 (local address 127.12.157.254:34629)
0810 07:25:02.402568 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:02.402573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:02.403556 (+ 983us) server_negotiation.cc:407] Connection header received
0810 07:25:02.404623 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:02.404628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:02.404695 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:02.404776 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:02.406350 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.407250 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.408568 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.408829 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.408976 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:02.412084 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:02.412088 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:02.412091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:02.412125 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:02.413861 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:02.414517 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:02.414521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:02.414523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:02.414617 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:02.415032 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:02.415037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:02.415040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:02.415268 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:02.415376 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:02.415875 (+ 499us) server_negotiation.cc:299] Negotiation successful
0810 07:25:02.416006 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 07:25:02.416347 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 14887
I20260810 07:25:02.416465 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance
I20260810 07:25:02.416702 12919 external_mini_cluster.cc:1428] Setting key 4108e4fbd4fe73d1b66be147c6a932cb
I20260810 07:25:02.417033 14929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:02.402886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:34629 (local address 127.12.157.193:47999)
0810 07:25:02.403365 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:02.403418 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:02.404359 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:02.405225 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:02.405233 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:02.405567 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:02.406169 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:02.406191 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.407417 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.407421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:02.408423 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:02.408434 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.409034 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.409039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:02.409233 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:02.409904 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:02.409927 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:02.411871 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:02.413983 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:02.413989 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:02.414002 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:02.414361 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:02.414756 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:02.414761 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:02.414764 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:02.414939 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:02.415377 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:02.415385 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:02.415633 (+ 248us) client_negotiation.cc:769] Sending connection context
0810 07:25:02.415829 (+ 196us) client_negotiation.cc:240] Negotiation successful
0810 07:25:02.416132 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":151,"threads_started":1}
I20260810 07:25:02.418743 15115 heartbeater.cc:347] Connected to a master server at 127.12.157.254:34629
I20260810 07:25:02.418942 15115 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:02.419566 15115 heartbeater.cc:511] Master 127.12.157.254:34629 requested a full tablet report, sending...
I20260810 07:25:02.421320 14775 ts_manager.cc:194] Registered new tserver with Master: e48974bc7ca24cf2b14a5b40c3861044 (127.12.157.193:46371)
I20260810 07:25:02.423044 14775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.193@KRBTEST.COM'} at 127.12.157.193:47999
WARNING: no policy specified for kudu/127.12.157.194@KRBTEST.COM; defaulting to no policy
I20260810 07:25:02.429181 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:02.421092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:60823 (local address 127.12.157.254:34629)
0810 07:25:02.421204 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:02.421209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:02.421410 (+ 201us) server_negotiation.cc:407] Connection header received
0810 07:25:02.421570 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:02.421573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:02.421615 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:02.421710 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:02.422167 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.422819 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.424008 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.424301 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.424410 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:02.425938 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:02.425940 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:02.425942 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:02.425963 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:02.427393 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:02.427902 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:02.427905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:02.427906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:02.427997 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:02.428328 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:02.428330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:02.428331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:02.428493 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:02.428570 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:02.428908 (+ 338us) server_negotiation.cc:299] Negotiation successful
0810 07:25:02.429063 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":35}
I20260810 07:25:02.429206 15127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:02.421025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:34629 (local address 127.12.157.193:60823)
0810 07:25:02.421312 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:02.421334 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:02.421451 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:02.421726 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:02.421729 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:02.421944 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:02.422047 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:02.422052 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.422948 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.422951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:02.423864 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:02.423879 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.424439 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.424447 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:02.424592 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:02.425223 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:02.425244 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:02.425786 (+ 542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:02.427512 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:02.427516 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:02.427518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:02.427786 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:02.428106 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:02.428112 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:02.428115 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:02.428219 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:02.428598 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:02.428604 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:02.428723 (+ 119us) client_negotiation.cc:769] Sending connection context
0810 07:25:02.428845 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 07:25:02.429028 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
I20260810 07:25:02.448853 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:0
--local_ip_for_outbound_sockets=127.12.157.194
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:34629
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:46815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:02.581995 15132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:02.582305 15132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:02.582356 15132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:02.586946 15132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:02.587020 15132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:02.587062 15132 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 07:25:02.587096 15132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:25:02.593394 15132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:46815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.157.254:34629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:02.594566 15132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:02.595614 15132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:02.605610 15145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:02.605600 15141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:02.605600 15138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:02.605821 15132 server_base.cc:1034] running on GCE node
I20260810 07:25:02.606341 15132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:02.607060 15132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:02.608269 15132 hybrid_clock.cc:648] HybridClock initialized: now 1786346702608248 us; error 34 us; skew 500 ppm
Aug 10 07:25:02 dist-test-slave-ffrd krb5kdc[14704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346702, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:02.611523 15132 init.cc:377] Logged in from keytab as kudu/127.12.157.194@KRBTEST.COM (short username kudu)
I20260810 07:25:02.612542 15132 webserver.cc:460] Webserver started at http://127.12.157.194:39659/ using document root <none> and password file <none>
I20260810 07:25:02.613250 15132 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:02.613348 15132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:02.613602 15132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:02.615720 15132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-1/data/instance:
uuid: "cb712788a82b466f978713691b6584b3"
format_stamp: "Formatted at 2026-08-10 07:25:02 on dist-test-slave-ffrd"
server_key: "a58d7290c179180fb1759d532b388835"
server_key_iv: "f77af3969bde2d188a738962140ba1d4"
server_key_version: "encryptionkey@0"
I20260810 07:25:02.616318 15132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance:
uuid: "cb712788a82b466f978713691b6584b3"
format_stamp: "Formatted at 2026-08-10 07:25:02 on dist-test-slave-ffrd"
server_key: "a58d7290c179180fb1759d532b388835"
server_key_iv: "f77af3969bde2d188a738962140ba1d4"
server_key_version: "encryptionkey@0"
I20260810 07:25:02.620615 15132 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260810 07:25:02.623822 15156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:02.625449 15132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:25:02.625618 15132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "cb712788a82b466f978713691b6584b3"
format_stamp: "Formatted at 2026-08-10 07:25:02 on dist-test-slave-ffrd"
server_key: "a58d7290c179180fb1759d532b388835"
server_key_iv: "f77af3969bde2d188a738962140ba1d4"
server_key_version: "encryptionkey@0"
I20260810 07:25:02.625764 15132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:02.644035 15132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:02.647864 15132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:02.648087 15132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:02.649761 15132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:02.649844 15132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:02.649940 15132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:02.649991 15132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 07:25:02 dist-test-slave-ffrd krb5kdc[14704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346702, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:02.662190 15132 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:42425
I20260810 07:25:02.662283 15357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:42425 every 8 connection(s)
I20260810 07:25:02.663134 15132 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:25:02.664872 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:02.650668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:58023 (local address 127.12.157.254:34629)
0810 07:25:02.650818 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:02.650823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:02.652015 (+ 1192us) server_negotiation.cc:407] Connection header received
0810 07:25:02.653239 (+ 1224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:02.653243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:02.653313 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:02.653421 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:02.654888 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.655529 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.656759 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.657158 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.657304 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:02.660825 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:02.660829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:02.660832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:02.660862 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:02.662628 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:02.663179 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:02.663183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:02.663185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:02.663280 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:02.663727 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:02.663730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:02.663732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:02.663983 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:02.664078 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:02.664623 (+ 545us) server_negotiation.cc:299] Negotiation successful
0810 07:25:02.664739 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 07:25:02.665738 15183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:02.651258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:34629 (local address 127.12.157.194:58023)
0810 07:25:02.651801 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:02.651880 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:02.652980 (+ 1100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:02.653631 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:02.653639 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:02.654024 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:02.654701 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:02.654720 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.655673 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.655677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:02.656613 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:02.656624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.657371 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.657376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:02.657632 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:02.658584 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:02.658605 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:02.660604 (+ 1999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:02.662759 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:02.662765 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:02.662781 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:02.663063 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:02.663386 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:02.663391 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:02.663396 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:02.663607 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:02.664105 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:02.664112 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:02.664387 (+ 275us) client_negotiation.cc:769] Sending connection context
0810 07:25:02.664556 (+ 169us) client_negotiation.cc:240] Negotiation successful
0810 07:25:02.664819 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":130,"threads_started":1}
I20260810 07:25:02.667204 15358 heartbeater.cc:347] Connected to a master server at 127.12.157.254:34629
I20260810 07:25:02.667380 15358 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:02.667944 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 15132
I20260810 07:25:02.668049 15358 heartbeater.cc:511] Master 127.12.157.254:34629 requested a full tablet report, sending...
I20260810 07:25:02.668077 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance
I20260810 07:25:02.668397 12919 external_mini_cluster.cc:1428] Setting key 8fa758baeb5332259b5fb7790112a21f
I20260810 07:25:02.669400 14773 ts_manager.cc:194] Registered new tserver with Master: cb712788a82b466f978713691b6584b3 (127.12.157.194:42425)
I20260810 07:25:02.670476 14773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.194@KRBTEST.COM'} at 127.12.157.194:58023
I20260810 07:25:02.678932 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:02.669720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:34629 (local address 127.12.157.194:55007)
0810 07:25:02.670127 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:02.670143 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:02.670252 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:02.670507 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:02.670511 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:02.670760 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:02.670912 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:02.670922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.671803 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.671806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:02.672700 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:02.672717 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.673270 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.673276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:02.673414 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:02.674044 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:02.674064 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:02.674494 (+ 430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:02.676918 (+ 2424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:02.676921 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:02.676923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:02.677213 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:02.677587 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:02.677589 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:02.677591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:02.677677 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:02.678342 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:02.678346 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:02.678528 (+ 182us) client_negotiation.cc:769] Sending connection context
0810 07:25:02.678630 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 07:25:02.678759 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":152,"threads_started":1}
I20260810 07:25:02.679200 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:02.669769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:55007 (local address 127.12.157.254:34629)
0810 07:25:02.669921 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:02.669926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:02.670224 (+ 298us) server_negotiation.cc:407] Connection header received
0810 07:25:02.670348 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:02.670350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:02.670389 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:02.670457 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:02.671044 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.671642 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.672840 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.673146 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.673236 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:02.674643 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:02.674649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:02.674654 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:02.674687 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:02.676753 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:02.677333 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:02.677341 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:02.677346 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:02.677460 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:02.677787 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:02.677794 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:02.677800 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:02.678198 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:02.678344 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:02.678853 (+ 509us) server_negotiation.cc:299] Negotiation successful
0810 07:25:02.679001 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.12.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
I20260810 07:25:02.705785 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:0
--local_ip_for_outbound_sockets=127.12.157.195
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:34629
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:46815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:02.829721 15374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:02.830035 15374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:02.830084 15374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:02.836639 15374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:02.836717 15374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:02.836763 15374 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 07:25:02.836800 15374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:25:02.842537 15374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:46815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.157.254:34629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:02.843626 15374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:02.844712 15374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:02.855593 15388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:02.855607 15383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:02.855626 15385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:02.855672 15374 server_base.cc:1034] running on GCE node
I20260810 07:25:02.856307 15374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:02.857031 15374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:02.858217 15374 hybrid_clock.cc:648] HybridClock initialized: now 1786346702858198 us; error 31 us; skew 500 ppm
Aug 10 07:25:02 dist-test-slave-ffrd krb5kdc[14704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346702, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:02.861227 15374 init.cc:377] Logged in from keytab as kudu/127.12.157.195@KRBTEST.COM (short username kudu)
I20260810 07:25:02.862145 15374 webserver.cc:460] Webserver started at http://127.12.157.195:42049/ using document root <none> and password file <none>
I20260810 07:25:02.862730 15374 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:02.862838 15374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:02.863088 15374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:02.865321 15374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-2/data/instance:
uuid: "fe564bfc9c7547c2a8dfe55bba612dbb"
format_stamp: "Formatted at 2026-08-10 07:25:02 on dist-test-slave-ffrd"
server_key: "90cddde9955085e74b7b7ff5c13683ec"
server_key_iv: "884f2d79152482523b429a54e0c2734b"
server_key_version: "encryptionkey@0"
I20260810 07:25:02.865911 15374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance:
uuid: "fe564bfc9c7547c2a8dfe55bba612dbb"
format_stamp: "Formatted at 2026-08-10 07:25:02 on dist-test-slave-ffrd"
server_key: "90cddde9955085e74b7b7ff5c13683ec"
server_key_iv: "884f2d79152482523b429a54e0c2734b"
server_key_version: "encryptionkey@0"
I20260810 07:25:02.870443 15374 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 07:25:02.873095 15399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:02.874451 15374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 07:25:02.874599 15374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "fe564bfc9c7547c2a8dfe55bba612dbb"
format_stamp: "Formatted at 2026-08-10 07:25:02 on dist-test-slave-ffrd"
server_key: "90cddde9955085e74b7b7ff5c13683ec"
server_key_iv: "884f2d79152482523b429a54e0c2734b"
server_key_version: "encryptionkey@0"
I20260810 07:25:02.874724 15374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:02.905055 15374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:02.910034 15374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:02.910282 15374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:02.912635 15374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:02.912840 15374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:02.913105 15374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:02.913223 15374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 07:25:02 dist-test-slave-ffrd krb5kdc[14704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346702, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:02.925436 15374 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:33133
I20260810 07:25:02.925491 15577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:33133 every 8 connection(s)
I20260810 07:25:02.926635 15374 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:25:02.928777 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:02.913092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:52035 (local address 127.12.157.254:34629)
0810 07:25:02.913324 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:02.913332 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:02.914291 (+ 959us) server_negotiation.cc:407] Connection header received
0810 07:25:02.915536 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:02.915540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:02.915633 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:02.915746 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:02.917416 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.918295 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.919977 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.920236 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.920365 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:02.924559 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:02.924563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:02.924567 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:02.924605 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:02.926308 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:02.926996 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:02.927000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:02.927001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:02.927137 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:02.927608 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:02.927613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:02.927614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:02.927874 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:02.927968 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:02.928536 (+ 568us) server_negotiation.cc:299] Negotiation successful
0810 07:25:02.928654 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 07:25:02.929723 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:02.913449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:34629 (local address 127.12.157.195:52035)
0810 07:25:02.914127 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:02.914180 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:02.915218 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:02.915941 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:02.915948 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:02.916461 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:02.917199 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:02.917224 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.918490 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.918496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:02.919753 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:02.919765 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.920429 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.920434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:02.920635 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:02.921418 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:02.921446 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:02.924299 (+ 2853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:02.926489 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:02.926495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:02.926509 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:02.926861 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:02.927266 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:02.927269 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:02.927271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:02.927466 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:02.927978 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:02.927984 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:02.928220 (+ 236us) client_negotiation.cc:769] Sending connection context
0810 07:25:02.928407 (+ 187us) client_negotiation.cc:240] Negotiation successful
0810 07:25:02.928680 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":173,"threads_started":1}
I20260810 07:25:02.931456 15579 heartbeater.cc:347] Connected to a master server at 127.12.157.254:34629
I20260810 07:25:02.931666 15579 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:02.932302 15579 heartbeater.cc:511] Master 127.12.157.254:34629 requested a full tablet report, sending...
I20260810 07:25:02.933544 14775 ts_manager.cc:194] Registered new tserver with Master: fe564bfc9c7547c2a8dfe55bba612dbb (127.12.157.195:33133)
I20260810 07:25:02.934278 14775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.195@KRBTEST.COM'} at 127.12.157.195:52035
I20260810 07:25:02.934963 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 15374
I20260810 07:25:02.935101 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance
I20260810 07:25:02.935436 12919 external_mini_cluster.cc:1428] Setting key bae7f7c3bf7aafcd615155dfeb1ca9c6
I20260810 07:25:02.937502 12919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:25:02.942947 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:02.934310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:54579 (local address 127.12.157.254:34629)
0810 07:25:02.934508 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:02.934512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:02.934787 (+ 275us) server_negotiation.cc:407] Connection header received
0810 07:25:02.934946 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:02.934949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:02.935009 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:02.935099 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:02.935700 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.936411 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.937611 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.937933 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.938052 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:02.939376 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:02.939382 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:02.939388 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:02.939422 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:02.941191 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:02.941663 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:02.941666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:02.941668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:02.941753 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:02.942064 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:02.942067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:02.942069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:02.942256 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:02.942372 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:02.942654 (+ 282us) server_negotiation.cc:299] Negotiation successful
0810 07:25:02.942769 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260810 07:25:02.942967 15585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:02.934219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:34629 (local address 127.12.157.195:54579)
0810 07:25:02.934691 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:02.934707 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:02.934836 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:02.935171 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:02.935174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:02.935417 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:02.935565 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:02.935573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.936651 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.936655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:02.937470 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:02.937481 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.938083 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.938086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:02.938168 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:02.938768 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:02.938781 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:02.939221 (+ 440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:02.941316 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:02.941319 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:02.941322 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:02.941549 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:02.941854 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:02.941857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:02.941860 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:02.941957 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:02.942361 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:02.942364 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:02.942478 (+ 114us) client_negotiation.cc:769] Sending connection context
0810 07:25:02.942594 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 07:25:02.942765 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":140,"threads_started":1}
Aug 10 07:25:02 dist-test-slave-ffrd krb5kdc[14704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346702, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 07:25:02 dist-test-slave-ffrd krb5kdc[14704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346702, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:02.969122 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:02.959057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48308 (local address 127.12.157.254:34629)
0810 07:25:02.959227 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:02.959231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:02.959244 (+ 13us) server_negotiation.cc:407] Connection header received
0810 07:25:02.959286 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:02.959288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:02.959335 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:02.959431 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:02.960189 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.960737 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.961669 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.961896 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.962006 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:02.964891 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:02.964894 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:02.964896 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:02.964925 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:02.966963 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:02.967632 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:02.967637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:02.967639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:02.967777 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:02.968129 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:02.968133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:02.968134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:02.968357 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:02.968506 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:02.968768 (+ 262us) server_negotiation.cc:299] Negotiation successful
0810 07:25:02.968917 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 07:25:02.972801 14775 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:25:02.975989 14775 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:25:02.992251 15601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:02.986460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54314 (local address 127.12.157.193:46371)
0810 07:25:02.986874 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:02.986882 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:02.987317 (+ 435us) server_negotiation.cc:407] Connection header received
0810 07:25:02.987434 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:02.987440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:02.987668 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:02.987825 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:02.988343 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.989405 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.990845 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.991559 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.991706 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:02.991853 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:02.992010 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 07:25:02.992109 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":68,"threads_started":1}
I20260810 07:25:02.992251 15599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:02.986239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:46371 (local address 127.0.0.1:54314)
0810 07:25:02.986952 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:02.987016 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:02.987165 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:02.987992 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:02.987997 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:02.988047 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:02.988203 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:02.988211 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.989533 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.989536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:02.990709 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:02.990718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.991692 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.991696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:02.991806 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:02.991844 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 07:25:02.991966 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 07:25:02.992096 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":162,"threads_started":1}
I20260810 07:25:02.993278 15602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:02.986908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:33133 (local address 127.0.0.1:38912)
0810 07:25:02.987701 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:02.987718 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:02.987797 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:02.989332 (+ 1535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:02.989334 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:02.989346 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:02.989425 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:02.989432 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.990736 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.990739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:02.991539 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:02.991550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.992838 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.992842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:02.992982 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:02.993008 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 07:25:02.993068 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 07:25:02.993137 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"mutex_wait_us":20,"thread_start_us":141,"threads_started":1}
I20260810 07:25:02.993352 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:02.987156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38912 (local address 127.12.157.195:33133)
0810 07:25:02.988807 (+ 1651us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:02.988818 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:02.988838 (+ 20us) server_negotiation.cc:407] Connection header received
0810 07:25:02.988908 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:02.988914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:02.989121 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:02.989375 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:02.989524 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.990615 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.991679 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.992689 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.992875 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:02.993035 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:02.993109 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 07:25:02.993201 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1526,"thread_start_us":125,"threads_started":1}
I20260810 07:25:02.994626 15606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:02.987892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:42425 (local address 127.0.0.1:36348)
0810 07:25:02.988142 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:02.988155 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:02.988238 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:02.989078 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:02.989081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:02.989101 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:02.989197 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:02.989203 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.990588 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.990591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:02.992724 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:02.992759 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.994229 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.994232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:02.994329 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:02.994352 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 07:25:02.994427 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 07:25:02.994496 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":1,"thread_start_us":104,"threads_started":1}
I20260810 07:25:02.995136 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:02.987906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36348 (local address 127.12.157.194:42425)
0810 07:25:02.988543 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:02.988580 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:02.988612 (+ 32us) server_negotiation.cc:407] Connection header received
0810 07:25:02.988696 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:02.988702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:02.988897 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:02.989104 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:02.989300 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.990448 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.992891 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:02.993874 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:02.994027 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:02.994156 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:02.994870 (+ 714us) server_negotiation.cc:299] Negotiation successful
0810 07:25:02.994979 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":125,"threads_started":1}
I20260810 07:25:02.995626 15004 tablet_service.cc:1467] Processing CreateTablet for tablet aa061305ab7f452993614c984fa8d1ee (DEFAULT_TABLE table=test-table [id=28526bea1499409bb6c23ae33d8c58f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:02.996234 15480 tablet_service.cc:1467] Processing CreateTablet for tablet aa061305ab7f452993614c984fa8d1ee (DEFAULT_TABLE table=test-table [id=28526bea1499409bb6c23ae33d8c58f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:02.996794 15004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa061305ab7f452993614c984fa8d1ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:02.997185 15480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa061305ab7f452993614c984fa8d1ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:02.997654 15251 tablet_service.cc:1467] Processing CreateTablet for tablet aa061305ab7f452993614c984fa8d1ee (DEFAULT_TABLE table=test-table [id=28526bea1499409bb6c23ae33d8c58f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:02.998611 15251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa061305ab7f452993614c984fa8d1ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:03.004309 15609 tablet_bootstrap.cc:492] T aa061305ab7f452993614c984fa8d1ee P fe564bfc9c7547c2a8dfe55bba612dbb: Bootstrap starting.
I20260810 07:25:03.005617 15610 tablet_bootstrap.cc:492] T aa061305ab7f452993614c984fa8d1ee P e48974bc7ca24cf2b14a5b40c3861044: Bootstrap starting.
I20260810 07:25:03.006129 15611 tablet_bootstrap.cc:492] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3: Bootstrap starting.
I20260810 07:25:03.007256 15609 tablet_bootstrap.cc:654] T aa061305ab7f452993614c984fa8d1ee P fe564bfc9c7547c2a8dfe55bba612dbb: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:03.007709 15610 tablet_bootstrap.cc:654] T aa061305ab7f452993614c984fa8d1ee P e48974bc7ca24cf2b14a5b40c3861044: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:03.008239 15609 log.cc:824] T aa061305ab7f452993614c984fa8d1ee P fe564bfc9c7547c2a8dfe55bba612dbb: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:03.008669 15611 tablet_bootstrap.cc:654] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:03.008680 15610 log.cc:824] T aa061305ab7f452993614c984fa8d1ee P e48974bc7ca24cf2b14a5b40c3861044: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:03.009604 15611 log.cc:824] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:03.010854 15609 tablet_bootstrap.cc:492] T aa061305ab7f452993614c984fa8d1ee P fe564bfc9c7547c2a8dfe55bba612dbb: No bootstrap required, opened a new log
I20260810 07:25:03.010854 15610 tablet_bootstrap.cc:492] T aa061305ab7f452993614c984fa8d1ee P e48974bc7ca24cf2b14a5b40c3861044: No bootstrap required, opened a new log
I20260810 07:25:03.011075 15610 ts_tablet_manager.cc:1397] T aa061305ab7f452993614c984fa8d1ee P e48974bc7ca24cf2b14a5b40c3861044: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260810 07:25:03.011075 15609 ts_tablet_manager.cc:1397] T aa061305ab7f452993614c984fa8d1ee P fe564bfc9c7547c2a8dfe55bba612dbb: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 07:25:03.011826 15611 tablet_bootstrap.cc:492] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3: No bootstrap required, opened a new log
I20260810 07:25:03.012255 15611 ts_tablet_manager.cc:1397] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260810 07:25:03.016768 15611 raft_consensus.cc:348] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } }
I20260810 07:25:03.016784 15609 raft_consensus.cc:348] T aa061305ab7f452993614c984fa8d1ee P fe564bfc9c7547c2a8dfe55bba612dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } }
I20260810 07:25:03.016785 15610 raft_consensus.cc:348] T aa061305ab7f452993614c984fa8d1ee P e48974bc7ca24cf2b14a5b40c3861044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } }
I20260810 07:25:03.017103 15609 raft_consensus.cc:374] T aa061305ab7f452993614c984fa8d1ee P fe564bfc9c7547c2a8dfe55bba612dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:03.017103 15610 raft_consensus.cc:374] T aa061305ab7f452993614c984fa8d1ee P e48974bc7ca24cf2b14a5b40c3861044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:03.017104 15611 raft_consensus.cc:374] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:03.017169 15611 raft_consensus.cc:729] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb712788a82b466f978713691b6584b3, State: Initialized, Role: FOLLOWER
I20260810 07:25:03.017170 15610 raft_consensus.cc:729] T aa061305ab7f452993614c984fa8d1ee P e48974bc7ca24cf2b14a5b40c3861044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e48974bc7ca24cf2b14a5b40c3861044, State: Initialized, Role: FOLLOWER
I20260810 07:25:03.017181 15609 raft_consensus.cc:729] T aa061305ab7f452993614c984fa8d1ee P fe564bfc9c7547c2a8dfe55bba612dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe564bfc9c7547c2a8dfe55bba612dbb, State: Initialized, Role: FOLLOWER
I20260810 07:25:03.017640 15611 consensus_queue.cc:260] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } }
I20260810 07:25:03.017653 15610 consensus_queue.cc:260] T aa061305ab7f452993614c984fa8d1ee P e48974bc7ca24cf2b14a5b40c3861044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } }
I20260810 07:25:03.017653 15609 consensus_queue.cc:260] T aa061305ab7f452993614c984fa8d1ee P fe564bfc9c7547c2a8dfe55bba612dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } }
I20260810 07:25:03.018623 15115 heartbeater.cc:503] Master 127.12.157.254:34629 was elected leader, sending a full tablet report...
I20260810 07:25:03.018864 15579 heartbeater.cc:503] Master 127.12.157.254:34629 was elected leader, sending a full tablet report...
I20260810 07:25:03.018972 15610 ts_tablet_manager.cc:1428] T aa061305ab7f452993614c984fa8d1ee P e48974bc7ca24cf2b14a5b40c3861044: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260810 07:25:03.019037 15609 ts_tablet_manager.cc:1428] T aa061305ab7f452993614c984fa8d1ee P fe564bfc9c7547c2a8dfe55bba612dbb: Time spent starting tablet: real 0.008s user 0.005s sys 0.005s
I20260810 07:25:03.020206 15611 ts_tablet_manager.cc:1428] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20260810 07:25:03.021231 15358 heartbeater.cc:503] Master 127.12.157.254:34629 was elected leader, sending a full tablet report...
I20260810 07:25:03.127277 15622 raft_consensus.cc:482] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:25:03.127516 15622 raft_consensus.cc:504] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } }
I20260810 07:25:03.128875 15622 leader_election.cc:288] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e48974bc7ca24cf2b14a5b40c3861044 (127.12.157.193:46371), fe564bfc9c7547c2a8dfe55bba612dbb (127.12.157.195:33133)
I20260810 07:25:03.133486 15183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:03.129255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:46371 (local address 127.12.157.194:50595)
0810 07:25:03.129516 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:03.129533 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:03.129642 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:03.129980 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:03.129985 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:03.130008 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:03.130156 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:03.130165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.131128 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.131131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:03.132349 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:03.132364 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.133154 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.133157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:03.133243 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:03.133263 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 07:25:03.133328 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 07:25:03.133383 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260810 07:25:03.133565 15601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:03.129397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:50595 (local address 127.12.157.193:46371)
0810 07:25:03.129538 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:03.129543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:03.129600 (+ 57us) server_negotiation.cc:407] Connection header received
0810 07:25:03.129764 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:03.129768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:03.129818 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:03.129916 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:03.130298 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.131011 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.132497 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.133006 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.133152 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:03.133216 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:03.133360 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 07:25:03.133441 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 07:25:03.134189 15624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:03.129259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:33133 (local address 127.12.157.194:42543)
0810 07:25:03.129981 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:03.129994 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:03.130090 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:03.130322 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:03.130325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:03.130343 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:03.130459 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:03.130466 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.131530 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.131533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:03.132997 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:03.133008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.133867 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.133873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:03.133958 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:03.133977 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 07:25:03.134038 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 07:25:03.134084 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":395,"thread_start_us":143,"threads_started":1}
I20260810 07:25:03.134287 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:03.129506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:42543 (local address 127.12.157.195:33133)
0810 07:25:03.129636 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:03.129641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:03.130086 (+ 445us) server_negotiation.cc:407] Connection header received
0810 07:25:03.130176 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:03.130179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:03.130231 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:03.130299 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:03.130601 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.131402 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.133119 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.133723 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.133853 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:03.133909 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:03.134081 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 07:25:03.134159 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 07:25:03.134222 15038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa061305ab7f452993614c984fa8d1ee" candidate_uuid: "cb712788a82b466f978713691b6584b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e48974bc7ca24cf2b14a5b40c3861044" is_pre_election: true
I20260810 07:25:03.134550 15038 raft_consensus.cc:2454] T aa061305ab7f452993614c984fa8d1ee P e48974bc7ca24cf2b14a5b40c3861044 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb712788a82b466f978713691b6584b3 in term 0.
I20260810 07:25:03.134909 15507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa061305ab7f452993614c984fa8d1ee" candidate_uuid: "cb712788a82b466f978713691b6584b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" is_pre_election: true
I20260810 07:25:03.135149 15159 leader_election.cc:302] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb712788a82b466f978713691b6584b3, e48974bc7ca24cf2b14a5b40c3861044; no voters:
I20260810 07:25:03.135181 15507 raft_consensus.cc:2454] T aa061305ab7f452993614c984fa8d1ee P fe564bfc9c7547c2a8dfe55bba612dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb712788a82b466f978713691b6584b3 in term 0.
I20260810 07:25:03.135432 15622 raft_consensus.cc:2781] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:25:03.135535 15622 raft_consensus.cc:482] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:25:03.135609 15622 raft_consensus.cc:3037] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:03.136710 15622 raft_consensus.cc:504] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } }
I20260810 07:25:03.137164 15622 leader_election.cc:288] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3 [CANDIDATE]: Term 1 election: Requested vote from peers e48974bc7ca24cf2b14a5b40c3861044 (127.12.157.193:46371), fe564bfc9c7547c2a8dfe55bba612dbb (127.12.157.195:33133)
I20260810 07:25:03.137570 15507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa061305ab7f452993614c984fa8d1ee" candidate_uuid: "cb712788a82b466f978713691b6584b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb"
I20260810 07:25:03.137590 15038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa061305ab7f452993614c984fa8d1ee" candidate_uuid: "cb712788a82b466f978713691b6584b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e48974bc7ca24cf2b14a5b40c3861044"
I20260810 07:25:03.137717 15038 raft_consensus.cc:3037] T aa061305ab7f452993614c984fa8d1ee P e48974bc7ca24cf2b14a5b40c3861044 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:03.137717 15507 raft_consensus.cc:3037] T aa061305ab7f452993614c984fa8d1ee P fe564bfc9c7547c2a8dfe55bba612dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:03.139230 15038 raft_consensus.cc:2454] T aa061305ab7f452993614c984fa8d1ee P e48974bc7ca24cf2b14a5b40c3861044 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb712788a82b466f978713691b6584b3 in term 1.
I20260810 07:25:03.139456 15507 raft_consensus.cc:2454] T aa061305ab7f452993614c984fa8d1ee P fe564bfc9c7547c2a8dfe55bba612dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb712788a82b466f978713691b6584b3 in term 1.
I20260810 07:25:03.139606 15159 leader_election.cc:302] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb712788a82b466f978713691b6584b3, e48974bc7ca24cf2b14a5b40c3861044; no voters:
I20260810 07:25:03.139786 15622 raft_consensus.cc:2781] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:03.140056 15622 raft_consensus.cc:686] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3 [term 1 LEADER]: Becoming Leader. State: Replica: cb712788a82b466f978713691b6584b3, State: Running, Role: LEADER
I20260810 07:25:03.140450 15622 consensus_queue.cc:237] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } }
I20260810 07:25:03.143834 14775 catalog_manager.cc:5495] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb712788a82b466f978713691b6584b3 (127.12.157.194). New cstate: current_term: 1 leader_uuid: "cb712788a82b466f978713691b6584b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } health_report { overall_health: HEALTHY } } }
W20260810 07:25:03.164577 15360 tablet.cc:2369] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 07:25:03.165740 15116 tablet.cc:2369] T aa061305ab7f452993614c984fa8d1ee P e48974bc7ca24cf2b14a5b40c3861044: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 07:25:03.177906 15580 tablet.cc:2369] T aa061305ab7f452993614c984fa8d1ee P fe564bfc9c7547c2a8dfe55bba612dbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:25:03.192008 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:03.188052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36350 (local address 127.12.157.194:42425)
0810 07:25:03.188218 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:03.188222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:03.188237 (+ 15us) server_negotiation.cc:407] Connection header received
0810 07:25:03.188507 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:03.188510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:03.188569 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:03.188699 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:25:03.189029 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.189626 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.190569 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.190777 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.190860 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:03.191199 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:25:03.191646 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:25:03.191743 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:03.191837 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 07:25:03.191890 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 07:25:03.198168 15038 raft_consensus.cc:1261] T aa061305ab7f452993614c984fa8d1ee P e48974bc7ca24cf2b14a5b40c3861044 [term 1 FOLLOWER]: Refusing update from remote peer cb712788a82b466f978713691b6584b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:03.198200 15507 raft_consensus.cc:1261] T aa061305ab7f452993614c984fa8d1ee P fe564bfc9c7547c2a8dfe55bba612dbb [term 1 FOLLOWER]: Refusing update from remote peer cb712788a82b466f978713691b6584b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:03.198938 15625 consensus_queue.cc:1035] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:03.199187 15622 consensus_queue.cc:1035] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:03.213559 15628 mvcc.cc:205] Tried to move back new op lower bound from 7316876096291348480 to 7316876096066785280. Current Snapshot: MvccSnapshot[applied={T|T < 7316876096291348480 or (T in {7316876096291348480})}]
I20260810 07:25:03.221668 14775 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:03.221902 14775 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:03.224463 14775 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69c4220dc13d4817a4497084935f8707: Sending DeleteTablet for 3 replicas of tablet aa061305ab7f452993614c984fa8d1ee
I20260810 07:25:03.225421 15004 tablet_service.cc:1514] Processing DeleteTablet for tablet aa061305ab7f452993614c984fa8d1ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:03 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:54314
I20260810 07:25:03.225692 15251 tablet_service.cc:1514] Processing DeleteTablet for tablet aa061305ab7f452993614c984fa8d1ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:03 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:36348
I20260810 07:25:03.226028 15480 tablet_service.cc:1514] Processing DeleteTablet for tablet aa061305ab7f452993614c984fa8d1ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:03 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:38912
I20260810 07:25:03.226150 15642 tablet_replica.cc:330] T aa061305ab7f452993614c984fa8d1ee P e48974bc7ca24cf2b14a5b40c3861044: stopping tablet replica
I20260810 07:25:03.226429 15644 tablet_replica.cc:330] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3: stopping tablet replica
I20260810 07:25:03.226540 15642 raft_consensus.cc:2229] T aa061305ab7f452993614c984fa8d1ee P e48974bc7ca24cf2b14a5b40c3861044 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:25:03.226858 15644 raft_consensus.cc:2229] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:25:03.226898 15642 raft_consensus.cc:2258] T aa061305ab7f452993614c984fa8d1ee P e48974bc7ca24cf2b14a5b40c3861044 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:03.227313 15644 raft_consensus.cc:2258] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:03.227409 14775 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:25:03.227871 14775 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:25:03.228559 15642 ts_tablet_manager.cc:1905] T aa061305ab7f452993614c984fa8d1ee P e48974bc7ca24cf2b14a5b40c3861044: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:03.228745 15647 tablet_replica.cc:330] T aa061305ab7f452993614c984fa8d1ee P fe564bfc9c7547c2a8dfe55bba612dbb: stopping tablet replica
I20260810 07:25:03.229086 15647 raft_consensus.cc:2229] T aa061305ab7f452993614c984fa8d1ee P fe564bfc9c7547c2a8dfe55bba612dbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:25:03.229439 15647 raft_consensus.cc:2258] T aa061305ab7f452993614c984fa8d1ee P fe564bfc9c7547c2a8dfe55bba612dbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:03.232040 15642 ts_tablet_manager.cc:1918] T aa061305ab7f452993614c984fa8d1ee P e48974bc7ca24cf2b14a5b40c3861044: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:25:03.232147 15642 log.cc:1196] T aa061305ab7f452993614c984fa8d1ee P e48974bc7ca24cf2b14a5b40c3861044: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-0/wal/wals/aa061305ab7f452993614c984fa8d1ee
I20260810 07:25:03.232484 15642 ts_tablet_manager.cc:1939] T aa061305ab7f452993614c984fa8d1ee P e48974bc7ca24cf2b14a5b40c3861044: Deleting consensus metadata
I20260810 07:25:03.232647 15644 ts_tablet_manager.cc:1905] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:03.233942 15480 tablet_service.cc:1467] Processing CreateTablet for tablet 1cee87eb5be742a8ad19f1fc8330b9a3 (DEFAULT_TABLE table=test-table [id=ef15c34b882349909485f86cba03ac71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:03.234236 15480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cee87eb5be742a8ad19f1fc8330b9a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:03.235267 15644 ts_tablet_manager.cc:1918] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:25:03.235387 15644 log.cc:1196] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-1/wal/wals/aa061305ab7f452993614c984fa8d1ee
I20260810 07:25:03.235670 15644 ts_tablet_manager.cc:1939] T aa061305ab7f452993614c984fa8d1ee P cb712788a82b466f978713691b6584b3: Deleting consensus metadata
I20260810 07:25:03.236935 15609 tablet_bootstrap.cc:492] T 1cee87eb5be742a8ad19f1fc8330b9a3 P fe564bfc9c7547c2a8dfe55bba612dbb: Bootstrap starting.
I20260810 07:25:03.237402 14750 catalog_manager.cc:4841] TS cb712788a82b466f978713691b6584b3 (127.12.157.194:42425): tablet aa061305ab7f452993614c984fa8d1ee (table test-table [id=28526bea1499409bb6c23ae33d8c58f0]) successfully deleted
I20260810 07:25:03.237563 14750 catalog_manager.cc:4841] TS e48974bc7ca24cf2b14a5b40c3861044 (127.12.157.193:46371): tablet aa061305ab7f452993614c984fa8d1ee (table test-table [id=28526bea1499409bb6c23ae33d8c58f0]) successfully deleted
I20260810 07:25:03.237630 15251 tablet_service.cc:1467] Processing CreateTablet for tablet 1cee87eb5be742a8ad19f1fc8330b9a3 (DEFAULT_TABLE table=test-table [id=ef15c34b882349909485f86cba03ac71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:03.237892 15251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cee87eb5be742a8ad19f1fc8330b9a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:03.237993 15004 tablet_service.cc:1467] Processing CreateTablet for tablet 1cee87eb5be742a8ad19f1fc8330b9a3 (DEFAULT_TABLE table=test-table [id=ef15c34b882349909485f86cba03ac71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:03.238086 15609 tablet_bootstrap.cc:654] T 1cee87eb5be742a8ad19f1fc8330b9a3 P fe564bfc9c7547c2a8dfe55bba612dbb: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:03.238233 15004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cee87eb5be742a8ad19f1fc8330b9a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:03.239486 15609 tablet_bootstrap.cc:492] T 1cee87eb5be742a8ad19f1fc8330b9a3 P fe564bfc9c7547c2a8dfe55bba612dbb: No bootstrap required, opened a new log
I20260810 07:25:03.239570 15609 ts_tablet_manager.cc:1397] T 1cee87eb5be742a8ad19f1fc8330b9a3 P fe564bfc9c7547c2a8dfe55bba612dbb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 07:25:03.240058 15609 raft_consensus.cc:348] T 1cee87eb5be742a8ad19f1fc8330b9a3 P fe564bfc9c7547c2a8dfe55bba612dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } }
I20260810 07:25:03.240175 15609 raft_consensus.cc:374] T 1cee87eb5be742a8ad19f1fc8330b9a3 P fe564bfc9c7547c2a8dfe55bba612dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:03.240212 15609 raft_consensus.cc:729] T 1cee87eb5be742a8ad19f1fc8330b9a3 P fe564bfc9c7547c2a8dfe55bba612dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe564bfc9c7547c2a8dfe55bba612dbb, State: Initialized, Role: FOLLOWER
I20260810 07:25:03.240262 15611 tablet_bootstrap.cc:492] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3: Bootstrap starting.
I20260810 07:25:03.240345 15609 consensus_queue.cc:260] T 1cee87eb5be742a8ad19f1fc8330b9a3 P fe564bfc9c7547c2a8dfe55bba612dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } }
I20260810 07:25:03.240598 15609 ts_tablet_manager.cc:1428] T 1cee87eb5be742a8ad19f1fc8330b9a3 P fe564bfc9c7547c2a8dfe55bba612dbb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 07:25:03.241068 15610 tablet_bootstrap.cc:492] T 1cee87eb5be742a8ad19f1fc8330b9a3 P e48974bc7ca24cf2b14a5b40c3861044: Bootstrap starting.
I20260810 07:25:03.242075 15610 tablet_bootstrap.cc:654] T 1cee87eb5be742a8ad19f1fc8330b9a3 P e48974bc7ca24cf2b14a5b40c3861044: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:03.242266 15647 ts_tablet_manager.cc:1905] T aa061305ab7f452993614c984fa8d1ee P fe564bfc9c7547c2a8dfe55bba612dbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:03.243125 15611 tablet_bootstrap.cc:654] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:03.244481 15610 tablet_bootstrap.cc:492] T 1cee87eb5be742a8ad19f1fc8330b9a3 P e48974bc7ca24cf2b14a5b40c3861044: No bootstrap required, opened a new log
I20260810 07:25:03.244585 15610 ts_tablet_manager.cc:1397] T 1cee87eb5be742a8ad19f1fc8330b9a3 P e48974bc7ca24cf2b14a5b40c3861044: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260810 07:25:03.245239 15611 tablet_bootstrap.cc:492] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3: No bootstrap required, opened a new log
I20260810 07:25:03.245337 15611 ts_tablet_manager.cc:1397] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260810 07:25:03.245543 15610 raft_consensus.cc:348] T 1cee87eb5be742a8ad19f1fc8330b9a3 P e48974bc7ca24cf2b14a5b40c3861044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } }
I20260810 07:25:03.245652 15610 raft_consensus.cc:374] T 1cee87eb5be742a8ad19f1fc8330b9a3 P e48974bc7ca24cf2b14a5b40c3861044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:03.245687 15610 raft_consensus.cc:729] T 1cee87eb5be742a8ad19f1fc8330b9a3 P e48974bc7ca24cf2b14a5b40c3861044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e48974bc7ca24cf2b14a5b40c3861044, State: Initialized, Role: FOLLOWER
I20260810 07:25:03.245826 15610 consensus_queue.cc:260] T 1cee87eb5be742a8ad19f1fc8330b9a3 P e48974bc7ca24cf2b14a5b40c3861044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } }
I20260810 07:25:03.245826 15611 raft_consensus.cc:348] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } }
I20260810 07:25:03.245915 15611 raft_consensus.cc:374] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:03.245949 15611 raft_consensus.cc:729] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb712788a82b466f978713691b6584b3, State: Initialized, Role: FOLLOWER
I20260810 07:25:03.246059 15611 consensus_queue.cc:260] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } }
I20260810 07:25:03.246119 15610 ts_tablet_manager.cc:1428] T 1cee87eb5be742a8ad19f1fc8330b9a3 P e48974bc7ca24cf2b14a5b40c3861044: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 07:25:03.246347 15611 ts_tablet_manager.cc:1428] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 07:25:03.246989 15647 ts_tablet_manager.cc:1918] T aa061305ab7f452993614c984fa8d1ee P fe564bfc9c7547c2a8dfe55bba612dbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:25:03.247084 15647 log.cc:1196] T aa061305ab7f452993614c984fa8d1ee P fe564bfc9c7547c2a8dfe55bba612dbb: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-2/wal/wals/aa061305ab7f452993614c984fa8d1ee
I20260810 07:25:03.247442 15647 ts_tablet_manager.cc:1939] T aa061305ab7f452993614c984fa8d1ee P fe564bfc9c7547c2a8dfe55bba612dbb: Deleting consensus metadata
I20260810 07:25:03.248694 14749 catalog_manager.cc:4841] TS fe564bfc9c7547c2a8dfe55bba612dbb (127.12.157.195:33133): tablet aa061305ab7f452993614c984fa8d1ee (table test-table [id=28526bea1499409bb6c23ae33d8c58f0]) successfully deleted
I20260810 07:25:03.337983 15636 raft_consensus.cc:482] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:25:03.338168 15636 raft_consensus.cc:504] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } }
I20260810 07:25:03.338858 15636 leader_election.cc:288] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e48974bc7ca24cf2b14a5b40c3861044 (127.12.157.193:46371), fe564bfc9c7547c2a8dfe55bba612dbb (127.12.157.195:33133)
I20260810 07:25:03.339428 15038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cee87eb5be742a8ad19f1fc8330b9a3" candidate_uuid: "cb712788a82b466f978713691b6584b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e48974bc7ca24cf2b14a5b40c3861044" is_pre_election: true
I20260810 07:25:03.339530 15508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cee87eb5be742a8ad19f1fc8330b9a3" candidate_uuid: "cb712788a82b466f978713691b6584b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" is_pre_election: true
I20260810 07:25:03.339608 15038 raft_consensus.cc:2454] T 1cee87eb5be742a8ad19f1fc8330b9a3 P e48974bc7ca24cf2b14a5b40c3861044 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb712788a82b466f978713691b6584b3 in term 0.
I20260810 07:25:03.339705 15508 raft_consensus.cc:2454] T 1cee87eb5be742a8ad19f1fc8330b9a3 P fe564bfc9c7547c2a8dfe55bba612dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb712788a82b466f978713691b6584b3 in term 0.
I20260810 07:25:03.340078 15157 leader_election.cc:302] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb712788a82b466f978713691b6584b3, fe564bfc9c7547c2a8dfe55bba612dbb; no voters:
I20260810 07:25:03.340299 15636 raft_consensus.cc:2781] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:25:03.340402 15636 raft_consensus.cc:482] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:25:03.340463 15636 raft_consensus.cc:3037] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:03.341872 15636 raft_consensus.cc:504] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } }
I20260810 07:25:03.342264 15636 leader_election.cc:288] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3 [CANDIDATE]: Term 1 election: Requested vote from peers e48974bc7ca24cf2b14a5b40c3861044 (127.12.157.193:46371), fe564bfc9c7547c2a8dfe55bba612dbb (127.12.157.195:33133)
I20260810 07:25:03.342836 15038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cee87eb5be742a8ad19f1fc8330b9a3" candidate_uuid: "cb712788a82b466f978713691b6584b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e48974bc7ca24cf2b14a5b40c3861044"
I20260810 07:25:03.342854 15508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cee87eb5be742a8ad19f1fc8330b9a3" candidate_uuid: "cb712788a82b466f978713691b6584b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb"
I20260810 07:25:03.342994 15508 raft_consensus.cc:3037] T 1cee87eb5be742a8ad19f1fc8330b9a3 P fe564bfc9c7547c2a8dfe55bba612dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:03.343001 15038 raft_consensus.cc:3037] T 1cee87eb5be742a8ad19f1fc8330b9a3 P e48974bc7ca24cf2b14a5b40c3861044 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:03.344282 15038 raft_consensus.cc:2454] T 1cee87eb5be742a8ad19f1fc8330b9a3 P e48974bc7ca24cf2b14a5b40c3861044 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb712788a82b466f978713691b6584b3 in term 1.
I20260810 07:25:03.344463 15508 raft_consensus.cc:2454] T 1cee87eb5be742a8ad19f1fc8330b9a3 P fe564bfc9c7547c2a8dfe55bba612dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb712788a82b466f978713691b6584b3 in term 1.
I20260810 07:25:03.344691 15159 leader_election.cc:302] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb712788a82b466f978713691b6584b3, e48974bc7ca24cf2b14a5b40c3861044; no voters:
I20260810 07:25:03.344967 15636 raft_consensus.cc:2781] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:03.345094 15636 raft_consensus.cc:686] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3 [term 1 LEADER]: Becoming Leader. State: Replica: cb712788a82b466f978713691b6584b3, State: Running, Role: LEADER
I20260810 07:25:03.345258 15636 consensus_queue.cc:237] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } }
I20260810 07:25:03.347108 14775 catalog_manager.cc:5495] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb712788a82b466f978713691b6584b3 (127.12.157.194). New cstate: current_term: 1 leader_uuid: "cb712788a82b466f978713691b6584b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } health_report { overall_health: UNKNOWN } } }
Aug 10 07:25:03 dist-test-slave-ffrd krb5kdc[14704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346702, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:03.394552 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:03.382964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48322 (local address 127.12.157.254:34629)
0810 07:25:03.383132 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:03.383137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:03.383479 (+ 342us) server_negotiation.cc:407] Connection header received
0810 07:25:03.383638 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:03.383641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:03.383683 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:03.383772 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:03.384704 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.385314 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.386821 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.387034 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.387118 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:03.390714 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:03.390717 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:03.390719 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:03.390741 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:03.392236 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:03.393089 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:03.393093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:03.393094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:03.393178 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:03.393509 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:03.393513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:03.393515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:03.393727 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:03.393857 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:03.394216 (+ 359us) server_negotiation.cc:299] Negotiation successful
0810 07:25:03.394413 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":58}
I20260810 07:25:03.394560 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:03.382845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:34629 (local address 127.0.0.1:48322)
0810 07:25:03.383348 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:03.383367 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:03.383526 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:03.383822 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:03.383826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:03.384447 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:03.384578 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:03.384587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.385453 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.385457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:03.386677 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:03.386689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.387171 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.387175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:03.387302 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:03.388176 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:03.388191 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:03.390560 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:03.392361 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:03.392371 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:03.392378 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:03.392902 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:03.393284 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:03.393288 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:03.393290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:03.393391 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:03.393877 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:03.393885 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:03.394037 (+ 152us) client_negotiation.cc:769] Sending connection context
0810 07:25:03.394161 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 07:25:03.394330 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":129,"threads_started":1}
I20260810 07:25:03.399163 14775 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:48322:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 07:25:03.399987 14775 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:25:03.405339 15004 tablet_service.cc:1467] Processing CreateTablet for tablet e356b90e325e4a0a9356d4bd11547971 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fd84503bedb34e7681a73d285ea46c0c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 07:25:03.405329 15251 tablet_service.cc:1467] Processing CreateTablet for tablet e356b90e325e4a0a9356d4bd11547971 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fd84503bedb34e7681a73d285ea46c0c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 07:25:03.405386 15480 tablet_service.cc:1467] Processing CreateTablet for tablet e356b90e325e4a0a9356d4bd11547971 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fd84503bedb34e7681a73d285ea46c0c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 07:25:03.405776 15480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e356b90e325e4a0a9356d4bd11547971. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:03.405776 15251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e356b90e325e4a0a9356d4bd11547971. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:03.405782 15004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e356b90e325e4a0a9356d4bd11547971. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:03.408773 15611 tablet_bootstrap.cc:492] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3: Bootstrap starting.
I20260810 07:25:03.408829 15610 tablet_bootstrap.cc:492] T e356b90e325e4a0a9356d4bd11547971 P e48974bc7ca24cf2b14a5b40c3861044: Bootstrap starting.
I20260810 07:25:03.410169 15611 tablet_bootstrap.cc:654] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:03.410463 15609 tablet_bootstrap.cc:492] T e356b90e325e4a0a9356d4bd11547971 P fe564bfc9c7547c2a8dfe55bba612dbb: Bootstrap starting.
I20260810 07:25:03.410583 15610 tablet_bootstrap.cc:654] T e356b90e325e4a0a9356d4bd11547971 P e48974bc7ca24cf2b14a5b40c3861044: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:03.412159 15609 tablet_bootstrap.cc:654] T e356b90e325e4a0a9356d4bd11547971 P fe564bfc9c7547c2a8dfe55bba612dbb: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:03.412245 15611 tablet_bootstrap.cc:492] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3: No bootstrap required, opened a new log
I20260810 07:25:03.412302 15610 tablet_bootstrap.cc:492] T e356b90e325e4a0a9356d4bd11547971 P e48974bc7ca24cf2b14a5b40c3861044: No bootstrap required, opened a new log
I20260810 07:25:03.412343 15611 ts_tablet_manager.cc:1397] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 07:25:03.412401 15610 ts_tablet_manager.cc:1397] T e356b90e325e4a0a9356d4bd11547971 P e48974bc7ca24cf2b14a5b40c3861044: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260810 07:25:03.413021 15611 raft_consensus.cc:348] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } }
I20260810 07:25:03.413151 15611 raft_consensus.cc:374] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:03.413193 15611 raft_consensus.cc:729] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb712788a82b466f978713691b6584b3, State: Initialized, Role: FOLLOWER
I20260810 07:25:03.413159 15610 raft_consensus.cc:348] T e356b90e325e4a0a9356d4bd11547971 P e48974bc7ca24cf2b14a5b40c3861044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } }
I20260810 07:25:03.413262 15610 raft_consensus.cc:374] T e356b90e325e4a0a9356d4bd11547971 P e48974bc7ca24cf2b14a5b40c3861044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:03.413316 15610 raft_consensus.cc:729] T e356b90e325e4a0a9356d4bd11547971 P e48974bc7ca24cf2b14a5b40c3861044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e48974bc7ca24cf2b14a5b40c3861044, State: Initialized, Role: FOLLOWER
I20260810 07:25:03.413360 15611 consensus_queue.cc:260] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } }
I20260810 07:25:03.413450 15610 consensus_queue.cc:260] T e356b90e325e4a0a9356d4bd11547971 P e48974bc7ca24cf2b14a5b40c3861044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } }
I20260810 07:25:03.413645 15636 tablet_replica.cc:439] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } } }
I20260810 07:25:03.413834 15636 tablet_replica.cc:442] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:25:03.413892 15610 ts_tablet_manager.cc:1428] T e356b90e325e4a0a9356d4bd11547971 P e48974bc7ca24cf2b14a5b40c3861044: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 07:25:03.414014 15609 tablet_bootstrap.cc:492] T e356b90e325e4a0a9356d4bd11547971 P fe564bfc9c7547c2a8dfe55bba612dbb: No bootstrap required, opened a new log
I20260810 07:25:03.414096 15609 ts_tablet_manager.cc:1397] T e356b90e325e4a0a9356d4bd11547971 P fe564bfc9c7547c2a8dfe55bba612dbb: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260810 07:25:03.414165 15619 tablet_replica.cc:439] T e356b90e325e4a0a9356d4bd11547971 P e48974bc7ca24cf2b14a5b40c3861044: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } } }
I20260810 07:25:03.414403 15619 tablet_replica.cc:442] T e356b90e325e4a0a9356d4bd11547971 P e48974bc7ca24cf2b14a5b40c3861044: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:25:03.414708 15609 raft_consensus.cc:348] T e356b90e325e4a0a9356d4bd11547971 P fe564bfc9c7547c2a8dfe55bba612dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } }
I20260810 07:25:03.414882 15609 raft_consensus.cc:374] T e356b90e325e4a0a9356d4bd11547971 P fe564bfc9c7547c2a8dfe55bba612dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:03.414915 15609 raft_consensus.cc:729] T e356b90e325e4a0a9356d4bd11547971 P fe564bfc9c7547c2a8dfe55bba612dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe564bfc9c7547c2a8dfe55bba612dbb, State: Initialized, Role: FOLLOWER
I20260810 07:25:03.415042 15609 consensus_queue.cc:260] T e356b90e325e4a0a9356d4bd11547971 P fe564bfc9c7547c2a8dfe55bba612dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } }
I20260810 07:25:03.415297 15609 ts_tablet_manager.cc:1428] T e356b90e325e4a0a9356d4bd11547971 P fe564bfc9c7547c2a8dfe55bba612dbb: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260810 07:25:03.415690 15615 tablet_replica.cc:439] T e356b90e325e4a0a9356d4bd11547971 P fe564bfc9c7547c2a8dfe55bba612dbb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } } }
I20260810 07:25:03.415850 15615 tablet_replica.cc:442] T e356b90e325e4a0a9356d4bd11547971 P fe564bfc9c7547c2a8dfe55bba612dbb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:25:03.416121 15611 ts_tablet_manager.cc:1428] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260810 07:25:03.418882 15636 raft_consensus.cc:482] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:25:03.418998 15636 raft_consensus.cc:504] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } }
I20260810 07:25:03.419384 15636 leader_election.cc:288] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e48974bc7ca24cf2b14a5b40c3861044 (127.12.157.193:46371), fe564bfc9c7547c2a8dfe55bba612dbb (127.12.157.195:33133)
I20260810 07:25:03.419732 15508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e356b90e325e4a0a9356d4bd11547971" candidate_uuid: "cb712788a82b466f978713691b6584b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" is_pre_election: true
I20260810 07:25:03.419765 15038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e356b90e325e4a0a9356d4bd11547971" candidate_uuid: "cb712788a82b466f978713691b6584b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e48974bc7ca24cf2b14a5b40c3861044" is_pre_election: true
I20260810 07:25:03.419896 15038 raft_consensus.cc:2454] T e356b90e325e4a0a9356d4bd11547971 P e48974bc7ca24cf2b14a5b40c3861044 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb712788a82b466f978713691b6584b3 in term 0.
I20260810 07:25:03.419891 15508 raft_consensus.cc:2454] T e356b90e325e4a0a9356d4bd11547971 P fe564bfc9c7547c2a8dfe55bba612dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb712788a82b466f978713691b6584b3 in term 0.
I20260810 07:25:03.420178 15159 leader_election.cc:302] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb712788a82b466f978713691b6584b3, e48974bc7ca24cf2b14a5b40c3861044; no voters:
I20260810 07:25:03.420387 15636 raft_consensus.cc:2781] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:25:03.420449 15636 raft_consensus.cc:482] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:25:03.420490 15636 raft_consensus.cc:3037] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:03.421937 15636 raft_consensus.cc:504] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } }
I20260810 07:25:03.422353 15636 leader_election.cc:288] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3 [CANDIDATE]: Term 1 election: Requested vote from peers e48974bc7ca24cf2b14a5b40c3861044 (127.12.157.193:46371), fe564bfc9c7547c2a8dfe55bba612dbb (127.12.157.195:33133)
I20260810 07:25:03.422796 15508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e356b90e325e4a0a9356d4bd11547971" candidate_uuid: "cb712788a82b466f978713691b6584b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb"
I20260810 07:25:03.423008 15508 raft_consensus.cc:3037] T e356b90e325e4a0a9356d4bd11547971 P fe564bfc9c7547c2a8dfe55bba612dbb [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:03.423041 15038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e356b90e325e4a0a9356d4bd11547971" candidate_uuid: "cb712788a82b466f978713691b6584b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e48974bc7ca24cf2b14a5b40c3861044"
I20260810 07:25:03.423142 15038 raft_consensus.cc:3037] T e356b90e325e4a0a9356d4bd11547971 P e48974bc7ca24cf2b14a5b40c3861044 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:03.424548 15508 raft_consensus.cc:2454] T e356b90e325e4a0a9356d4bd11547971 P fe564bfc9c7547c2a8dfe55bba612dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb712788a82b466f978713691b6584b3 in term 1.
I20260810 07:25:03.424548 15038 raft_consensus.cc:2454] T e356b90e325e4a0a9356d4bd11547971 P e48974bc7ca24cf2b14a5b40c3861044 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb712788a82b466f978713691b6584b3 in term 1.
I20260810 07:25:03.424840 15157 leader_election.cc:302] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb712788a82b466f978713691b6584b3, fe564bfc9c7547c2a8dfe55bba612dbb; no voters:
I20260810 07:25:03.425033 15636 raft_consensus.cc:2781] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:03.425120 15636 raft_consensus.cc:686] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3 [term 1 LEADER]: Becoming Leader. State: Replica: cb712788a82b466f978713691b6584b3, State: Running, Role: LEADER
I20260810 07:25:03.425239 15636 consensus_queue.cc:237] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } }
I20260810 07:25:03.425933 15625 tablet_replica.cc:439] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3: TxnStatusTablet state changed. Reason: New leader cb712788a82b466f978713691b6584b3. Latest consensus state: current_term: 1 leader_uuid: "cb712788a82b466f978713691b6584b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } } }
I20260810 07:25:03.426026 15625 tablet_replica.cc:442] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3: This TxnStatusTablet replica's current role is: LEADER
I20260810 07:25:03.426690 15674 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 07:25:03.426903 15674 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 07:25:03.427170 14775 catalog_manager.cc:5495] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb712788a82b466f978713691b6584b3 (127.12.157.194). New cstate: current_term: 1 leader_uuid: "cb712788a82b466f978713691b6584b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } health_report { overall_health: UNKNOWN } } }
I20260810 07:25:03.442415 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:03.438326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:42425 (local address 127.0.0.1:36354)
0810 07:25:03.438627 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:03.438640 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:03.438739 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:03.439052 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:03.439055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:03.439062 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 07:25:03.439291 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:03.439296 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.440294 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.440297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:03.440916 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:03.440924 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.441631 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.441634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:03.441746 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:03.441794 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 07:25:03.442125 (+ 331us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 07:25:03.442142 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 07:25:03.442213 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 07:25:03.442296 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199}
I20260810 07:25:03.442432 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:03.438474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36354 (local address 127.12.157.194:42425)
0810 07:25:03.438599 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:03.438604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:03.438707 (+ 103us) server_negotiation.cc:407] Connection header received
0810 07:25:03.438849 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:03.438853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:03.438905 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:03.438988 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:25:03.439400 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.440175 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.441180 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.441518 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.441622 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:03.441925 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:25:03.442008 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:25:03.442082 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:03.442244 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 07:25:03.442297 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 07:25:03.445353 15508 raft_consensus.cc:1261] T e356b90e325e4a0a9356d4bd11547971 P fe564bfc9c7547c2a8dfe55bba612dbb [term 1 FOLLOWER]: Refusing update from remote peer cb712788a82b466f978713691b6584b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:03.445353 15038 raft_consensus.cc:1261] T e356b90e325e4a0a9356d4bd11547971 P e48974bc7ca24cf2b14a5b40c3861044 [term 1 FOLLOWER]: Refusing update from remote peer cb712788a82b466f978713691b6584b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:03.446296 15625 consensus_queue.cc:1035] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:03.446645 15625 consensus_queue.cc:1035] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:03.448341 15615 tablet_replica.cc:439] T e356b90e325e4a0a9356d4bd11547971 P fe564bfc9c7547c2a8dfe55bba612dbb: TxnStatusTablet state changed. Reason: New leader cb712788a82b466f978713691b6584b3. Latest consensus state: current_term: 1 leader_uuid: "cb712788a82b466f978713691b6584b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } } }
I20260810 07:25:03.448472 15615 tablet_replica.cc:442] T e356b90e325e4a0a9356d4bd11547971 P fe564bfc9c7547c2a8dfe55bba612dbb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:25:03.448421 15619 tablet_replica.cc:439] T e356b90e325e4a0a9356d4bd11547971 P e48974bc7ca24cf2b14a5b40c3861044: TxnStatusTablet state changed. Reason: New leader cb712788a82b466f978713691b6584b3. Latest consensus state: current_term: 1 leader_uuid: "cb712788a82b466f978713691b6584b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } } }
I20260810 07:25:03.448530 15619 tablet_replica.cc:442] T e356b90e325e4a0a9356d4bd11547971 P e48974bc7ca24cf2b14a5b40c3861044: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:25:03.450214 15636 tablet_replica.cc:439] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cb712788a82b466f978713691b6584b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } } }
I20260810 07:25:03.450347 15636 tablet_replica.cc:442] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3: This TxnStatusTablet replica's current role is: LEADER
I20260810 07:25:03.450769 15615 tablet_replica.cc:439] T e356b90e325e4a0a9356d4bd11547971 P fe564bfc9c7547c2a8dfe55bba612dbb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cb712788a82b466f978713691b6584b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } } }
I20260810 07:25:03.450909 15615 tablet_replica.cc:442] T e356b90e325e4a0a9356d4bd11547971 P fe564bfc9c7547c2a8dfe55bba612dbb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:25:03.453408 15630 mvcc.cc:205] Tried to move back new op lower bound from 7316876097307521024 to 7316876097231204352. Current Snapshot: MvccSnapshot[applied={T|T < 7316876097307521024 or (T in {7316876097307521024})}]
I20260810 07:25:03.454635 15625 tablet_replica.cc:439] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cb712788a82b466f978713691b6584b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } } }
I20260810 07:25:03.454756 15625 tablet_replica.cc:442] T e356b90e325e4a0a9356d4bd11547971 P cb712788a82b466f978713691b6584b3: This TxnStatusTablet replica's current role is: LEADER
I20260810 07:25:03.455510 15619 tablet_replica.cc:439] T e356b90e325e4a0a9356d4bd11547971 P e48974bc7ca24cf2b14a5b40c3861044: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cb712788a82b466f978713691b6584b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 } } peers { permanent_uuid: "cb712788a82b466f978713691b6584b3" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 42425 } } peers { permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 } } }
I20260810 07:25:03.455632 15619 tablet_replica.cc:442] T e356b90e325e4a0a9356d4bd11547971 P e48974bc7ca24cf2b14a5b40c3861044: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:25:03.470940 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:03.467074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:44089 (local address 127.12.157.194:42425)
0810 07:25:03.467208 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:03.467213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:03.467543 (+ 330us) server_negotiation.cc:407] Connection header received
0810 07:25:03.467709 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:03.467714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:03.467766 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:03.467847 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:25:03.468157 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.468719 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.469786 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.469997 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.470107 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:03.470384 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:25:03.470483 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:25:03.470577 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:03.470667 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 07:25:03.470775 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":45}
I20260810 07:25:03.470940 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:03.466970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:42425 (local address 127.12.157.194:44089)
0810 07:25:03.467455 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:03.467468 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:03.467606 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:03.467855 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:03.467858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:03.467869 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 07:25:03.468025 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:03.468031 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.468847 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.468854 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:03.469663 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:03.469674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.470108 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.470111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:03.470194 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:03.470234 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 07:25:03.470590 (+ 356us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 07:25:03.470597 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 07:25:03.470660 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 07:25:03.470788 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":18,"thread_start_us":181,"threads_started":1}
I20260810 07:25:03.478579 15038 raft_consensus.cc:1261] T 1cee87eb5be742a8ad19f1fc8330b9a3 P e48974bc7ca24cf2b14a5b40c3861044 [term 1 FOLLOWER]: Refusing update from remote peer cb712788a82b466f978713691b6584b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:03.478659 15507 raft_consensus.cc:1261] T 1cee87eb5be742a8ad19f1fc8330b9a3 P fe564bfc9c7547c2a8dfe55bba612dbb [term 1 FOLLOWER]: Refusing update from remote peer cb712788a82b466f978713691b6584b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:03.479104 15636 consensus_queue.cc:1035] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e48974bc7ca24cf2b14a5b40c3861044" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:03.479262 15622 consensus_queue.cc:1035] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe564bfc9c7547c2a8dfe55bba612dbb" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:03.524883 14773 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:03.525122 14773 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:03.527217 14773 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 69c4220dc13d4817a4497084935f8707: Sending DeleteTablet for 3 replicas of tablet 1cee87eb5be742a8ad19f1fc8330b9a3
I20260810 07:25:03.527869 15480 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cee87eb5be742a8ad19f1fc8330b9a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:03 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:38912
I20260810 07:25:03.527906 15004 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cee87eb5be742a8ad19f1fc8330b9a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:03 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:54314
I20260810 07:25:03.528070 15642 tablet_replica.cc:330] T 1cee87eb5be742a8ad19f1fc8330b9a3 P e48974bc7ca24cf2b14a5b40c3861044: stopping tablet replica
I20260810 07:25:03.528071 15647 tablet_replica.cc:330] T 1cee87eb5be742a8ad19f1fc8330b9a3 P fe564bfc9c7547c2a8dfe55bba612dbb: stopping tablet replica
I20260810 07:25:03.528039 15250 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cee87eb5be742a8ad19f1fc8330b9a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:03 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:36348
I20260810 07:25:03.528183 15647 raft_consensus.cc:2229] T 1cee87eb5be742a8ad19f1fc8330b9a3 P fe564bfc9c7547c2a8dfe55bba612dbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:25:03.528178 15642 raft_consensus.cc:2229] T 1cee87eb5be742a8ad19f1fc8330b9a3 P e48974bc7ca24cf2b14a5b40c3861044 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:25:03.528227 15644 tablet_replica.cc:330] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3: stopping tablet replica
I20260810 07:25:03.528288 15642 raft_consensus.cc:2258] T 1cee87eb5be742a8ad19f1fc8330b9a3 P e48974bc7ca24cf2b14a5b40c3861044 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:03.528288 15647 raft_consensus.cc:2258] T 1cee87eb5be742a8ad19f1fc8330b9a3 P fe564bfc9c7547c2a8dfe55bba612dbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:03.528344 15644 raft_consensus.cc:2229] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:25:03.528540 15644 raft_consensus.cc:2258] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:03.529271 15647 ts_tablet_manager.cc:1905] T 1cee87eb5be742a8ad19f1fc8330b9a3 P fe564bfc9c7547c2a8dfe55bba612dbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:03.529333 15644 ts_tablet_manager.cc:1905] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:03.529346 15642 ts_tablet_manager.cc:1905] T 1cee87eb5be742a8ad19f1fc8330b9a3 P e48974bc7ca24cf2b14a5b40c3861044: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:03.532140 15647 ts_tablet_manager.cc:1918] T 1cee87eb5be742a8ad19f1fc8330b9a3 P fe564bfc9c7547c2a8dfe55bba612dbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 07:25:03.532140 15644 ts_tablet_manager.cc:1918] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 07:25:03.532183 15642 ts_tablet_manager.cc:1918] T 1cee87eb5be742a8ad19f1fc8330b9a3 P e48974bc7ca24cf2b14a5b40c3861044: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 07:25:03.532213 15647 log.cc:1196] T 1cee87eb5be742a8ad19f1fc8330b9a3 P fe564bfc9c7547c2a8dfe55bba612dbb: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-2/wal/wals/1cee87eb5be742a8ad19f1fc8330b9a3
I20260810 07:25:03.532212 15644 log.cc:1196] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-1/wal/wals/1cee87eb5be742a8ad19f1fc8330b9a3
I20260810 07:25:03.532259 15642 log.cc:1196] T 1cee87eb5be742a8ad19f1fc8330b9a3 P e48974bc7ca24cf2b14a5b40c3861044: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786346694975122-12919-0/minicluster-data/ts-0/wal/wals/1cee87eb5be742a8ad19f1fc8330b9a3
I20260810 07:25:03.532495 15647 ts_tablet_manager.cc:1939] T 1cee87eb5be742a8ad19f1fc8330b9a3 P fe564bfc9c7547c2a8dfe55bba612dbb: Deleting consensus metadata
I20260810 07:25:03.532495 15644 ts_tablet_manager.cc:1939] T 1cee87eb5be742a8ad19f1fc8330b9a3 P cb712788a82b466f978713691b6584b3: Deleting consensus metadata
I20260810 07:25:03.532523 15642 ts_tablet_manager.cc:1939] T 1cee87eb5be742a8ad19f1fc8330b9a3 P e48974bc7ca24cf2b14a5b40c3861044: Deleting consensus metadata
I20260810 07:25:03.533373 14750 catalog_manager.cc:4841] TS cb712788a82b466f978713691b6584b3 (127.12.157.194:42425): tablet 1cee87eb5be742a8ad19f1fc8330b9a3 (table test-table [id=ef15c34b882349909485f86cba03ac71]) successfully deleted
I20260810 07:25:03.533566 14750 catalog_manager.cc:4841] TS e48974bc7ca24cf2b14a5b40c3861044 (127.12.157.193:46371): tablet 1cee87eb5be742a8ad19f1fc8330b9a3 (table test-table [id=ef15c34b882349909485f86cba03ac71]) successfully deleted
I20260810 07:25:03.533977 14749 catalog_manager.cc:4841] TS fe564bfc9c7547c2a8dfe55bba612dbb (127.12.157.195:33133): tablet 1cee87eb5be742a8ad19f1fc8330b9a3 (table test-table [id=ef15c34b882349909485f86cba03ac71]) successfully deleted
Aug 10 07:25:03 dist-test-slave-ffrd krb5kdc[14704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346702, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.157.193@KRBTEST.COM
I20260810 07:25:03.546072 15601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:03.533756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54328 (local address 127.12.157.193:46371)
0810 07:25:03.533904 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:03.533909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:03.534094 (+ 185us) server_negotiation.cc:407] Connection header received
0810 07:25:03.534269 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:03.534272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:03.534316 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:03.534375 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:03.535477 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.536019 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.537498 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.537745 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.537847 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:03.541979 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:03.541992 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:03.542008 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:03.542035 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:03.543818 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:03.544375 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:03.544381 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:03.544383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:03.544465 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:03.544798 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:03.544801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:03.544803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:03.545200 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:03.545454 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:03.545693 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 07:25:03.545898 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260810 07:25:03.546701 15107 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54328
I20260810 07:25:03.556404 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:03.548501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48326 (local address 127.12.157.254:34629)
0810 07:25:03.548636 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:03.548641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:03.548691 (+ 50us) server_negotiation.cc:407] Connection header received
0810 07:25:03.548818 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:03.548821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:03.548872 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:03.548976 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:03.549602 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.550428 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.551263 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:03.551597 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:03.551706 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:03.553172 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:03.553176 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:03.553179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:03.553206 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:03.554733 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:03.555165 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:03.555168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:03.555170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:03.555264 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:03.555531 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:03.555534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:03.555535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:03.555742 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:03.555869 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:03.556099 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 07:25:03.556213 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260810 07:25:03.556885 14773 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48326
I20260810 07:25:03.558423 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 14887
I20260810 07:25:03.577174 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 15132
I20260810 07:25:03.594163 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 15374
I20260810 07:25:03.608453 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 14722
2026-08-10T07:25:03Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3104 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:25:03 dist-test-slave-ffrd krb5kdc[15699](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:25:03 dist-test-slave-ffrd krb5kdc[15699](info): set up 2 sockets
Aug 10 07:25:03 dist-test-slave-ffrd krb5kdc[15699](info): commencing operation
krb5kdc: starting...
W20260810 07:25:04.906612 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.265s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:25:04 dist-test-slave-ffrd krb5kdc[15699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:25:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:25:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:25:05.041467 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:36711
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:36711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:05.166276 15724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:05.166599 15724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:05.166643 15724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:05.171504 15724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:25:05.171597 15724 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 07:25:05.171614 15724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:05.171634 15724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:05.171651 15724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:25:05.171729 15724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:25:05.177891 15724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:36711
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:36711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:05.179287 15724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:05.180392 15724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 07:25:05.189287 15724 server_base.cc:1034] running on GCE node
W20260810 07:25:05.189248 15734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:05.189272 15731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:05.189244 15732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:05.190124 15724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:05.191195 15724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:05.192368 15724 hybrid_clock.cc:648] HybridClock initialized: now 1786346705192347 us; error 32 us; skew 500 ppm
Aug 10 07:25:05 dist-test-slave-ffrd krb5kdc[15699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346705, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:05.198112 15724 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:25:05.199092 15724 webserver.cc:460] Webserver started at http://127.12.157.254:34635/ using document root <none> and password file <none>
I20260810 07:25:05.199678 15724 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:05.199743 15724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:05.199985 15724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:05.202207 15724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "5f64d8b2626d44b29f277990aa61bf4e"
format_stamp: "Formatted at 2026-08-10 07:25:05 on dist-test-slave-ffrd"
server_key: "21884cce34b00c4c22bca7ce6b7b0b05"
server_key_iv: "3005d26fc9b993d5c811eea8f4bde12a"
server_key_version: "encryptionkey@0"
I20260810 07:25:05.202782 15724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "5f64d8b2626d44b29f277990aa61bf4e"
format_stamp: "Formatted at 2026-08-10 07:25:05 on dist-test-slave-ffrd"
server_key: "21884cce34b00c4c22bca7ce6b7b0b05"
server_key_iv: "3005d26fc9b993d5c811eea8f4bde12a"
server_key_version: "encryptionkey@0"
I20260810 07:25:05.206808 15724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 07:25:05.210088 15744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:05.211436 15724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 07:25:05.211570 15724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "5f64d8b2626d44b29f277990aa61bf4e"
format_stamp: "Formatted at 2026-08-10 07:25:05 on dist-test-slave-ffrd"
server_key: "21884cce34b00c4c22bca7ce6b7b0b05"
server_key_iv: "3005d26fc9b993d5c811eea8f4bde12a"
server_key_version: "encryptionkey@0"
I20260810 07:25:05.211696 15724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:05.228287 15724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:05.231798 15724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:05.232014 15724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:05.240608 15724 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:36711
I20260810 07:25:05.240689 15830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:36711 every 8 connection(s)
I20260810 07:25:05.241820 15724 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:25:05.244822 15831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:05.250603 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 15724
I20260810 07:25:05.250711 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:25:05.250640 15831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f64d8b2626d44b29f277990aa61bf4e: Bootstrap starting.
I20260810 07:25:05.251014 12919 external_mini_cluster.cc:1428] Setting key 0ba266e41e9a266608968de44151212f
I20260810 07:25:05.253460 15831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f64d8b2626d44b29f277990aa61bf4e: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:05.254392 15831 log.cc:824] T 00000000000000000000000000000000 P 5f64d8b2626d44b29f277990aa61bf4e: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:05.256969 15831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f64d8b2626d44b29f277990aa61bf4e: No bootstrap required, opened a new log
I20260810 07:25:05.260991 15831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f64d8b2626d44b29f277990aa61bf4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f64d8b2626d44b29f277990aa61bf4e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 36711 } }
I20260810 07:25:05.261277 15831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f64d8b2626d44b29f277990aa61bf4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:05.261361 15831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f64d8b2626d44b29f277990aa61bf4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f64d8b2626d44b29f277990aa61bf4e, State: Initialized, Role: FOLLOWER
Aug 10 07:25:05 dist-test-slave-ffrd krb5kdc[15699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:05.261940 15831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f64d8b2626d44b29f277990aa61bf4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f64d8b2626d44b29f277990aa61bf4e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 36711 } }
I20260810 07:25:05.262115 15831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f64d8b2626d44b29f277990aa61bf4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:25:05.262200 15831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f64d8b2626d44b29f277990aa61bf4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:25:05.262302 15831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f64d8b2626d44b29f277990aa61bf4e [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:05.263310 15831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f64d8b2626d44b29f277990aa61bf4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f64d8b2626d44b29f277990aa61bf4e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 36711 } }
I20260810 07:25:05.263677 15831 leader_election.cc:302] T 00000000000000000000000000000000 P 5f64d8b2626d44b29f277990aa61bf4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f64d8b2626d44b29f277990aa61bf4e; no voters:
I20260810 07:25:05.264087 15831 leader_election.cc:288] T 00000000000000000000000000000000 P 5f64d8b2626d44b29f277990aa61bf4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:25:05.264160 15841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f64d8b2626d44b29f277990aa61bf4e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:05.264377 15841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f64d8b2626d44b29f277990aa61bf4e [term 1 LEADER]: Becoming Leader. State: Replica: 5f64d8b2626d44b29f277990aa61bf4e, State: Running, Role: LEADER
I20260810 07:25:05.264740 15841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f64d8b2626d44b29f277990aa61bf4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f64d8b2626d44b29f277990aa61bf4e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 36711 } }
I20260810 07:25:05.265515 15831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f64d8b2626d44b29f277990aa61bf4e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:25:05.266464 15842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f64d8b2626d44b29f277990aa61bf4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f64d8b2626d44b29f277990aa61bf4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f64d8b2626d44b29f277990aa61bf4e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 36711 } } }
I20260810 07:25:05.266608 15842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f64d8b2626d44b29f277990aa61bf4e [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:05.266930 15845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f64d8b2626d44b29f277990aa61bf4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f64d8b2626d44b29f277990aa61bf4e. Latest consensus state: current_term: 1 leader_uuid: "5f64d8b2626d44b29f277990aa61bf4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f64d8b2626d44b29f277990aa61bf4e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 36711 } } }
I20260810 07:25:05.267019 15845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f64d8b2626d44b29f277990aa61bf4e [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:05.268113 15864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:25:05.268316 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:05.252509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42542 (local address 127.12.157.254:36711)
0810 07:25:05.253019 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:05.253039 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:05.253081 (+ 42us) server_negotiation.cc:407] Connection header received
0810 07:25:05.254039 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:05.254064 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:05.254494 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:05.254880 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:05.255803 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.257273 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.258237 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.259057 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.259228 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:05.262238 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:05.262243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:05.262278 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:05.262316 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:05.264333 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:05.264938 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:05.264945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:05.264979 (+ 34us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:05.265102 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:05.265415 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:05.265419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:05.265421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:05.265961 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:05.266774 (+ 813us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:05.266941 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 07:25:05.267249 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":143,"threads_started":1}
I20260810 07:25:05.271696 15864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:25:05.278321 15864 catalog_manager.cc:1347] Generated new cluster ID: 1c1501bf63d94550aa9961a89726db3f
I20260810 07:25:05.278431 15864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:25:05.297736 15864 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:25:05.299017 15864 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:25:05.310881 15864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f64d8b2626d44b29f277990aa61bf4e: Generated new TSK 0
I20260810 07:25:05.311774 15864 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
I20260810 07:25:05.352280 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:0
--local_ip_for_outbound_sockets=127.12.157.193
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:36711
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:05.469696 15891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:05.469988 15891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:05.470039 15891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:05.474225 15891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:05.474408 15891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:05.474498 15891 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 07:25:05.474550 15891 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 07:25:05.474622 15891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:25:05.480314 15891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.157.254:36711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:05.481514 15891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:05.482712 15891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:05.497380 15899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:05.497320 15905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:05.497386 15901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:05.497745 15891 server_base.cc:1034] running on GCE node
I20260810 07:25:05.498353 15891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:05.499081 15891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:05.500305 15891 hybrid_clock.cc:648] HybridClock initialized: now 1786346705500271 us; error 55 us; skew 500 ppm
Aug 10 07:25:05 dist-test-slave-ffrd krb5kdc[15699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346705, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:05.503396 15891 init.cc:377] Logged in from keytab as kudu/127.12.157.193@KRBTEST.COM (short username kudu)
I20260810 07:25:05.504326 15891 webserver.cc:460] Webserver started at http://127.12.157.193:41939/ using document root <none> and password file <none>
I20260810 07:25:05.504900 15891 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:05.505023 15891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:05.505223 15891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:05.507308 15891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-0/data/instance:
uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e"
format_stamp: "Formatted at 2026-08-10 07:25:05 on dist-test-slave-ffrd"
server_key: "18fd213d2553c93a8cc20acf2fb09c5b"
server_key_iv: "9d7d8a00108374e3ffec5dc11f55b76f"
server_key_version: "encryptionkey@0"
I20260810 07:25:05.507838 15891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance:
uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e"
format_stamp: "Formatted at 2026-08-10 07:25:05 on dist-test-slave-ffrd"
server_key: "18fd213d2553c93a8cc20acf2fb09c5b"
server_key_iv: "9d7d8a00108374e3ffec5dc11f55b76f"
server_key_version: "encryptionkey@0"
I20260810 07:25:05.511816 15891 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260810 07:25:05.514528 15916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:05.516185 15891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 07:25:05.516335 15891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e"
format_stamp: "Formatted at 2026-08-10 07:25:05 on dist-test-slave-ffrd"
server_key: "18fd213d2553c93a8cc20acf2fb09c5b"
server_key_iv: "9d7d8a00108374e3ffec5dc11f55b76f"
server_key_version: "encryptionkey@0"
I20260810 07:25:05.516680 15891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:05.533627 15891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:05.537212 15891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:05.537432 15891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:05.538923 15891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:05.538998 15891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:05.539072 15891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:05.539106 15891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 07:25:05 dist-test-slave-ffrd krb5kdc[15699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346705, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:05.550719 15891 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:41553
I20260810 07:25:05.551615 15891 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:25:05.552456 16099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:41553 every 8 connection(s)
I20260810 07:25:05.553876 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:05.540046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:38493 (local address 127.12.157.254:36711)
0810 07:25:05.540182 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:05.540186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:05.541391 (+ 1205us) server_negotiation.cc:407] Connection header received
0810 07:25:05.542662 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:05.542666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:05.542728 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:05.542850 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:05.544575 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.545360 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.546678 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.546938 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.547055 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:05.549916 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:05.549920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:05.549923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:05.549952 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:05.551606 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:05.552218 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:05.552222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:05.552224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:05.552322 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:05.552753 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:05.552756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:05.552757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:05.553017 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:05.553117 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:05.553636 (+ 519us) server_negotiation.cc:299] Negotiation successful
0810 07:25:05.553747 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 07:25:05.554642 15931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:05.540636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:36711 (local address 127.12.157.193:38493)
0810 07:25:05.541221 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:05.541271 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:05.542444 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:05.543082 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:05.543095 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:05.543608 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:05.544386 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:05.544412 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.545501 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.545504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:05.546550 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:05.546561 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.547125 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.547130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:05.547297 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:05.547941 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:05.547958 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:05.549730 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:05.551775 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:05.551781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:05.551794 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:05.552085 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:05.552430 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:05.552433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:05.552435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:05.552628 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:05.553142 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:05.553147 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:05.553405 (+ 258us) client_negotiation.cc:769] Sending connection context
0810 07:25:05.553574 (+ 169us) client_negotiation.cc:240] Negotiation successful
0810 07:25:05.553822 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":136,"threads_started":1}
I20260810 07:25:05.555927 16100 heartbeater.cc:347] Connected to a master server at 127.12.157.254:36711
I20260810 07:25:05.556068 16100 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:05.556645 16100 heartbeater.cc:511] Master 127.12.157.254:36711 requested a full tablet report, sending...
I20260810 07:25:05.558382 15764 ts_manager.cc:194] Registered new tserver with Master: cfc9dd8eb7bb4fcc9068037c315efa1e (127.12.157.193:41553)
I20260810 07:25:05.559803 15764 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.193@KRBTEST.COM'} at 127.12.157.193:38493
I20260810 07:25:05.560039 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 15891
I20260810 07:25:05.560155 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance
I20260810 07:25:05.560490 12919 external_mini_cluster.cc:1428] Setting key 32d70b170f79e310a6e820e5059ab671
I20260810 07:25:05.566051 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:05.558370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:56557 (local address 127.12.157.254:36711)
0810 07:25:05.558520 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:05.558525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:05.558724 (+ 199us) server_negotiation.cc:407] Connection header received
0810 07:25:05.558814 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:05.558816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:05.558857 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:05.558930 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:05.559464 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.560158 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.561363 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.561582 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.561694 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:05.562892 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:05.562896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:05.562898 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:05.562928 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:05.564362 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:05.564796 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:05.564800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:05.564802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:05.564882 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:05.565210 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:05.565215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:05.565216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:05.565387 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:05.565502 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:05.565762 (+ 260us) server_negotiation.cc:299] Negotiation successful
0810 07:25:05.565873 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 07:25:05.566089 16109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:05.558284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:36711 (local address 127.12.157.193:56557)
0810 07:25:05.558612 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:05.558627 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:05.558732 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:05.558971 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:05.558974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:05.559201 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:05.559336 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:05.559343 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.560325 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.560329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:05.561222 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:05.561232 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.561699 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.561703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:05.561796 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:05.562363 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:05.562379 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:05.562756 (+ 377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:05.564483 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:05.564486 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:05.564489 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:05.564693 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:05.565019 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:05.565022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:05.565024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:05.565115 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:05.565491 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:05.565494 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:05.565596 (+ 102us) client_negotiation.cc:769] Sending connection context
0810 07:25:05.565709 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 07:25:05.565878 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.12.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
I20260810 07:25:05.592399 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:0
--local_ip_for_outbound_sockets=127.12.157.194
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:36711
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:05.713315 16113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:05.713579 16113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:05.713663 16113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:05.717988 16113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:05.718051 16113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:05.718094 16113 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 07:25:05.718144 16113 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 07:25:05.718199 16113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:25:05.723678 16113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.157.254:36711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:05.724783 16113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:05.725867 16113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:05.735709 16124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:05.735893 16113 server_base.cc:1034] running on GCE node
W20260810 07:25:05.735724 16123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:05.736055 16131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:05.736457 16113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:05.737169 16113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:05.738363 16113 hybrid_clock.cc:648] HybridClock initialized: now 1786346705738336 us; error 38 us; skew 500 ppm
Aug 10 07:25:05 dist-test-slave-ffrd krb5kdc[15699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346705, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:05.741430 16113 init.cc:377] Logged in from keytab as kudu/127.12.157.194@KRBTEST.COM (short username kudu)
I20260810 07:25:05.742372 16113 webserver.cc:460] Webserver started at http://127.12.157.194:37293/ using document root <none> and password file <none>
I20260810 07:25:05.742973 16113 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:05.743070 16113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:05.743309 16113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:05.745451 16113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-1/data/instance:
uuid: "d2c0e0687ac5456584fc4db0949c2f12"
format_stamp: "Formatted at 2026-08-10 07:25:05 on dist-test-slave-ffrd"
server_key: "4858f60055cb29b4565adb80323c7b95"
server_key_iv: "f3e42846f03e10609276859896f1ea9e"
server_key_version: "encryptionkey@0"
I20260810 07:25:05.746025 16113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance:
uuid: "d2c0e0687ac5456584fc4db0949c2f12"
format_stamp: "Formatted at 2026-08-10 07:25:05 on dist-test-slave-ffrd"
server_key: "4858f60055cb29b4565adb80323c7b95"
server_key_iv: "f3e42846f03e10609276859896f1ea9e"
server_key_version: "encryptionkey@0"
I20260810 07:25:05.750044 16113 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 07:25:05.752689 16141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:05.754222 16113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 07:25:05.754385 16113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "d2c0e0687ac5456584fc4db0949c2f12"
format_stamp: "Formatted at 2026-08-10 07:25:05 on dist-test-slave-ffrd"
server_key: "4858f60055cb29b4565adb80323c7b95"
server_key_iv: "f3e42846f03e10609276859896f1ea9e"
server_key_version: "encryptionkey@0"
I20260810 07:25:05.754519 16113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:05.780046 16113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:05.783861 16113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:05.784096 16113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:05.785984 16113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:05.786065 16113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:05.786166 16113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:05.786216 16113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 07:25:05 dist-test-slave-ffrd krb5kdc[15699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346705, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:05.798178 16113 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:46585
I20260810 07:25:05.799467 16113 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:25:05.799626 16318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:46585 every 8 connection(s)
I20260810 07:25:05.800392 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 16113
I20260810 07:25:05.800513 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance
I20260810 07:25:05.800812 12919 external_mini_cluster.cc:1428] Setting key 6272dc2a7fe1039e7c70f1aa181651bf
I20260810 07:25:05.801414 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:05.786705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:41751 (local address 127.12.157.254:36711)
0810 07:25:05.786857 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:05.786862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:05.787869 (+ 1007us) server_negotiation.cc:407] Connection header received
0810 07:25:05.789195 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:05.789216 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:05.789282 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:05.789381 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:05.791258 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.791858 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.793250 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.793494 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.793619 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:05.796769 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:05.796773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:05.796775 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:05.796799 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:05.798667 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:05.799191 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:05.799195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:05.799197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:05.799288 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:05.799679 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:05.799683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:05.799685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:05.799909 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:05.800005 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:05.801103 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0810 07:25:05.801242 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 07:25:05.801877 16157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:05.787147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:36711 (local address 127.12.157.194:41751)
0810 07:25:05.787677 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:05.787738 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:05.788914 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:05.789609 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:05.789619 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:05.790258 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:05.791023 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:05.791048 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.792017 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.792021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:05.793126 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:05.793136 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.793679 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.793684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:05.793851 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:05.794579 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:05.794595 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:05.796586 (+ 1991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:05.798778 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:05.798784 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:05.798800 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:05.799080 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:05.799401 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:05.799405 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:05.799406 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:05.799593 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:05.800003 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:05.800009 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:05.800333 (+ 324us) client_negotiation.cc:769] Sending connection context
0810 07:25:05.800553 (+ 220us) client_negotiation.cc:240] Negotiation successful
0810 07:25:05.800909 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":115,"threads_started":1}
I20260810 07:25:05.804363 16319 heartbeater.cc:347] Connected to a master server at 127.12.157.254:36711
I20260810 07:25:05.804545 16319 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:05.805176 16319 heartbeater.cc:511] Master 127.12.157.254:36711 requested a full tablet report, sending...
I20260810 07:25:05.806437 15764 ts_manager.cc:194] Registered new tserver with Master: d2c0e0687ac5456584fc4db0949c2f12 (127.12.157.194:46585)
I20260810 07:25:05.807088 15764 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.194@KRBTEST.COM'} at 127.12.157.194:41751
WARNING: no policy specified for kudu/127.12.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.195@KRBTEST.COM" created.
I20260810 07:25:05.816890 16330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:05.806610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:36711 (local address 127.12.157.194:34229)
0810 07:25:05.807271 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:05.807300 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:05.807438 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:05.809107 (+ 1669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:05.809109 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:05.809296 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:05.809397 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:05.809403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.810250 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.810255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:05.811271 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:05.811296 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.811790 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.811795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:05.811935 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:05.812521 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:05.812542 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:05.813073 (+ 531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:05.815087 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:05.815090 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:05.815091 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:05.815291 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:05.815618 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:05.815621 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:05.815622 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:05.815703 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:05.816231 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:05.816233 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:05.816316 (+ 83us) client_negotiation.cc:769] Sending connection context
0810 07:25:05.816425 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 07:25:05.816574 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":91,"threads_started":1}
I20260810 07:25:05.816931 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:05.806965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:34229 (local address 127.12.157.254:36711)
0810 07:25:05.807090 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:05.807096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:05.807367 (+ 271us) server_negotiation.cc:407] Connection header received
0810 07:25:05.807543 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:05.807548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:05.807602 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:05.807656 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:05.809523 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.810088 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.811441 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:05.811655 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:05.811740 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:05.813212 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:05.813216 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:05.813218 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:05.813240 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:05.814983 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:05.815426 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:05.815429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:05.815430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:05.815534 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:05.815832 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:05.815835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:05.815837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:05.816130 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:05.816214 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:05.816630 (+ 416us) server_negotiation.cc:299] Negotiation successful
0810 07:25:05.816798 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
I20260810 07:25:05.837116 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:0
--local_ip_for_outbound_sockets=127.12.157.195
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:36711
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:05.962556 16334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:05.962828 16334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:05.962877 16334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:05.966967 16334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:05.967026 16334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:05.967067 16334 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 07:25:05.967080 16334 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 07:25:05.967166 16334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:25:05.973213 16334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.157.254:36711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:05.974354 16334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:05.975458 16334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:05.985862 16341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:05.985862 16348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:05.985945 16340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:05.986111 16334 server_base.cc:1034] running on GCE node
I20260810 07:25:05.986724 16334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:05.987421 16334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:05.988641 16334 hybrid_clock.cc:648] HybridClock initialized: now 1786346705988606 us; error 49 us; skew 500 ppm
Aug 10 07:25:05 dist-test-slave-ffrd krb5kdc[15699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346705, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:05.991811 16334 init.cc:377] Logged in from keytab as kudu/127.12.157.195@KRBTEST.COM (short username kudu)
I20260810 07:25:05.992759 16334 webserver.cc:460] Webserver started at http://127.12.157.195:34929/ using document root <none> and password file <none>
I20260810 07:25:05.993498 16334 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:05.993625 16334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:05.993884 16334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:05.996023 16334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-2/data/instance:
uuid: "f7c8cbc8254d40fba61eaa0984a3fe47"
format_stamp: "Formatted at 2026-08-10 07:25:05 on dist-test-slave-ffrd"
server_key: "e22fc0cfe834a6b78a0688b78268a2ed"
server_key_iv: "51f18f4d2af47d73b953a6a2238a8deb"
server_key_version: "encryptionkey@0"
I20260810 07:25:05.996773 16334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance:
uuid: "f7c8cbc8254d40fba61eaa0984a3fe47"
format_stamp: "Formatted at 2026-08-10 07:25:05 on dist-test-slave-ffrd"
server_key: "e22fc0cfe834a6b78a0688b78268a2ed"
server_key_iv: "51f18f4d2af47d73b953a6a2238a8deb"
server_key_version: "encryptionkey@0"
I20260810 07:25:06.000443 16334 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 07:25:06.002974 16355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:06.004405 16334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 07:25:06.004558 16334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "f7c8cbc8254d40fba61eaa0984a3fe47"
format_stamp: "Formatted at 2026-08-10 07:25:05 on dist-test-slave-ffrd"
server_key: "e22fc0cfe834a6b78a0688b78268a2ed"
server_key_iv: "51f18f4d2af47d73b953a6a2238a8deb"
server_key_version: "encryptionkey@0"
I20260810 07:25:06.004683 16334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:06.016249 16334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:06.019408 16334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:06.019622 16334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:06.021314 16334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:06.021391 16334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:06.021466 16334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:06.021518 16334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 07:25:06 dist-test-slave-ffrd krb5kdc[15699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346705, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:06.033322 16334 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:34729
I20260810 07:25:06.033397 16529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:34729 every 8 connection(s)
I20260810 07:25:06.034396 16334 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:25:06.035815 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.022175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:55259 (local address 127.12.157.254:36711)
0810 07:25:06.022366 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:06.022373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:06.023197 (+ 824us) server_negotiation.cc:407] Connection header received
0810 07:25:06.024251 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:06.024255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:06.024321 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:06.024413 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:06.026064 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.026624 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.028415 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.028757 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.028895 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.031516 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:06.031520 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:06.031523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:06.031558 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:06.033427 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:06.033969 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:06.033973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:06.033975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:06.034072 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:06.034479 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:06.034483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:06.034484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:06.034724 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:06.034846 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:06.035493 (+ 647us) server_negotiation.cc:299] Negotiation successful
0810 07:25:06.035629 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 07:25:06.035838 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 16334
I20260810 07:25:06.035946 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance
I20260810 07:25:06.036252 12919 external_mini_cluster.cc:1428] Setting key c805eae5c21e8c9da02ca29da84288c7
I20260810 07:25:06.036660 16370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.022550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:36711 (local address 127.12.157.195:55259)
0810 07:25:06.023022 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:06.023073 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:06.023974 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:06.024798 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:06.024809 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:06.025291 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:06.025915 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.025932 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.026763 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.026766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.028293 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.028307 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.028979 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.028985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.029175 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.029812 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:06.029831 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:06.031341 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:06.033555 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:06.033561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:06.033577 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:06.033859 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:06.034202 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:06.034205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:06.034206 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:06.034359 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:06.034860 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:06.034867 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:06.035180 (+ 313us) client_negotiation.cc:769] Sending connection context
0810 07:25:06.035392 (+ 212us) client_negotiation.cc:240] Negotiation successful
0810 07:25:06.035651 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":151,"threads_started":1}
I20260810 07:25:06.038030 16530 heartbeater.cc:347] Connected to a master server at 127.12.157.254:36711
I20260810 07:25:06.038226 16530 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:06.038870 16530 heartbeater.cc:511] Master 127.12.157.254:36711 requested a full tablet report, sending...
I20260810 07:25:06.039940 15764 ts_manager.cc:194] Registered new tserver with Master: f7c8cbc8254d40fba61eaa0984a3fe47 (127.12.157.195:34729)
I20260810 07:25:06.040539 15764 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.195@KRBTEST.COM'} at 127.12.157.195:55259
I20260810 07:25:06.048029 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.041109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:53623 (local address 127.12.157.254:36711)
0810 07:25:06.041236 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:06.041241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:06.041432 (+ 191us) server_negotiation.cc:407] Connection header received
0810 07:25:06.041520 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:06.041523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:06.041570 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:06.041655 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:06.042232 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.042905 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.043716 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.043943 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.044024 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.045145 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:06.045147 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:06.045150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:06.045172 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:06.046675 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:06.047065 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:06.047067 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:06.047068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:06.047141 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:06.047387 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:06.047389 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:06.047390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:06.047546 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:06.047622 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:06.047817 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 07:25:06.047904 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 07:25:06.048055 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.041076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:36711 (local address 127.12.157.195:53623)
0810 07:25:06.041331 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:06.041346 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:06.041454 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:06.041679 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:06.041681 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:06.041972 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:06.042108 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.042115 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.043056 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.043059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.043609 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.043618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.044044 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.044046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.044114 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.044599 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:06.044611 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:06.045014 (+ 403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:06.046779 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:06.046781 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:06.046783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:06.046980 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:06.047224 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:06.047226 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:06.047227 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:06.047304 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:06.047628 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:06.047631 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:06.047705 (+ 74us) client_negotiation.cc:769] Sending connection context
0810 07:25:06.047787 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 07:25:06.047912 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":99,"threads_started":1}
I20260810 07:25:06.050525 12919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 07:25:06 dist-test-slave-ffrd krb5kdc[15699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 07:25:06 dist-test-slave-ffrd krb5kdc[15699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:06.078603 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.069908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42558 (local address 127.12.157.254:36711)
0810 07:25:06.070050 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:06.070054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:06.070066 (+ 12us) server_negotiation.cc:407] Connection header received
0810 07:25:06.070107 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:06.070109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:06.070147 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:06.070242 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:06.070983 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.071521 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.072381 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.072600 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.072707 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.075330 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:06.075333 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:06.075335 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:06.075367 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:06.076838 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:06.077310 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:06.077314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:06.077315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:06.077401 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:06.077691 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:06.077695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:06.077697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:06.077928 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:06.078073 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:06.078308 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 07:25:06.078426 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 07:25:06.081646 15764 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:25:06.084631 15764 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:25:06.099444 16563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.094278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:41553 (local address 127.0.0.1:36656)
0810 07:25:06.094602 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:06.094616 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:06.094696 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:06.095570 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:06.095573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:06.095596 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:06.095703 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.095709 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.096970 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.096973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.098073 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.098084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.099122 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.099128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.099212 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.099232 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 07:25:06.099286 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 07:25:06.099343 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":22,"thread_start_us":97,"threads_started":1}
I20260810 07:25:06.099586 16564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.094471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36656 (local address 127.12.157.193:41553)
0810 07:25:06.095158 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:06.095165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:06.095189 (+ 24us) server_negotiation.cc:407] Connection header received
0810 07:25:06.095264 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:06.095270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:06.095439 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:06.095591 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:06.095824 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.096821 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.098213 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.098964 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.099135 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.099259 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:06.099330 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 07:25:06.099434 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":85,"threads_started":1}
I20260810 07:25:06.099849 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.093714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37718 (local address 127.12.157.195:34729)
0810 07:25:06.094035 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:06.094048 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:06.094075 (+ 27us) server_negotiation.cc:407] Connection header received
0810 07:25:06.094212 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:06.094220 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:06.094440 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:06.094610 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:06.095135 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.096308 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.098019 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.098718 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.098898 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.099044 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:06.099513 (+ 469us) server_negotiation.cc:299] Negotiation successful
0810 07:25:06.099614 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":127,"threads_started":1}
I20260810 07:25:06.100055 16566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.094599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:46585 (local address 127.0.0.1:58630)
0810 07:25:06.094820 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:06.094833 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:06.094925 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:06.095920 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:06.095923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:06.095942 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:06.096017 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.096023 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.097137 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.097140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.098294 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.098303 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.099621 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.099627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.099726 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.099758 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 07:25:06.099854 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 07:25:06.099926 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":22,"thread_start_us":76,"threads_started":1}
I20260810 07:25:06.100131 16567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.095042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58630 (local address 127.12.157.194:46585)
0810 07:25:06.095469 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:06.095480 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:06.095502 (+ 22us) server_negotiation.cc:407] Connection header received
0810 07:25:06.095571 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:06.095575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:06.095736 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:06.095935 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:06.096138 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.097018 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.098442 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.099252 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.099428 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.099547 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:06.099889 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 07:25:06.099991 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":110,"threads_started":1}
I20260810 07:25:06.099444 16559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.093543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:34729 (local address 127.0.0.1:37718)
0810 07:25:06.093889 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:06.093933 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:06.094056 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:06.094806 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:06.094813 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:06.094834 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:06.094990 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.094997 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.096428 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.096432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.097874 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.097886 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.098865 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.098869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.098990 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.099029 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 07:25:06.099139 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 07:25:06.099258 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":2,"thread_start_us":116,"threads_started":1}
I20260810 07:25:06.102926 15999 tablet_service.cc:1467] Processing CreateTablet for tablet a0c2f91618ed4e88855c1c76a3754502 (DEFAULT_TABLE table=test-table [id=3a5d12521bce4f58a9c0d0b3b63bedf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:06.103019 16218 tablet_service.cc:1467] Processing CreateTablet for tablet a0c2f91618ed4e88855c1c76a3754502 (DEFAULT_TABLE table=test-table [id=3a5d12521bce4f58a9c0d0b3b63bedf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:06.103950 15999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0c2f91618ed4e88855c1c76a3754502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:06.104085 16218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0c2f91618ed4e88855c1c76a3754502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:06.103729 16427 tablet_service.cc:1467] Processing CreateTablet for tablet a0c2f91618ed4e88855c1c76a3754502 (DEFAULT_TABLE table=test-table [id=3a5d12521bce4f58a9c0d0b3b63bedf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:06.104736 16427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0c2f91618ed4e88855c1c76a3754502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:06.110342 16568 tablet_bootstrap.cc:492] T a0c2f91618ed4e88855c1c76a3754502 P cfc9dd8eb7bb4fcc9068037c315efa1e: Bootstrap starting.
I20260810 07:25:06.112161 16571 tablet_bootstrap.cc:492] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12: Bootstrap starting.
I20260810 07:25:06.112641 16568 tablet_bootstrap.cc:654] T a0c2f91618ed4e88855c1c76a3754502 P cfc9dd8eb7bb4fcc9068037c315efa1e: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:06.113157 16573 tablet_bootstrap.cc:492] T a0c2f91618ed4e88855c1c76a3754502 P f7c8cbc8254d40fba61eaa0984a3fe47: Bootstrap starting.
I20260810 07:25:06.114113 16568 log.cc:824] T a0c2f91618ed4e88855c1c76a3754502 P cfc9dd8eb7bb4fcc9068037c315efa1e: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:06.114449 16571 tablet_bootstrap.cc:654] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:06.115291 16573 tablet_bootstrap.cc:654] T a0c2f91618ed4e88855c1c76a3754502 P f7c8cbc8254d40fba61eaa0984a3fe47: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:06.115262 16571 log.cc:824] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:06.116170 16568 tablet_bootstrap.cc:492] T a0c2f91618ed4e88855c1c76a3754502 P cfc9dd8eb7bb4fcc9068037c315efa1e: No bootstrap required, opened a new log
I20260810 07:25:06.116196 16573 log.cc:824] T a0c2f91618ed4e88855c1c76a3754502 P f7c8cbc8254d40fba61eaa0984a3fe47: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:06.116402 16568 ts_tablet_manager.cc:1397] T a0c2f91618ed4e88855c1c76a3754502 P cfc9dd8eb7bb4fcc9068037c315efa1e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260810 07:25:06.117559 16571 tablet_bootstrap.cc:492] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12: No bootstrap required, opened a new log
I20260810 07:25:06.117769 16571 ts_tablet_manager.cc:1397] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 07:25:06.118134 16573 tablet_bootstrap.cc:492] T a0c2f91618ed4e88855c1c76a3754502 P f7c8cbc8254d40fba61eaa0984a3fe47: No bootstrap required, opened a new log
I20260810 07:25:06.118331 16573 ts_tablet_manager.cc:1397] T a0c2f91618ed4e88855c1c76a3754502 P f7c8cbc8254d40fba61eaa0984a3fe47: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 07:25:06.121428 16568 raft_consensus.cc:348] T a0c2f91618ed4e88855c1c76a3754502 P cfc9dd8eb7bb4fcc9068037c315efa1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } } peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } }
I20260810 07:25:06.121721 16568 raft_consensus.cc:374] T a0c2f91618ed4e88855c1c76a3754502 P cfc9dd8eb7bb4fcc9068037c315efa1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:06.121837 16568 raft_consensus.cc:729] T a0c2f91618ed4e88855c1c76a3754502 P cfc9dd8eb7bb4fcc9068037c315efa1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc9dd8eb7bb4fcc9068037c315efa1e, State: Initialized, Role: FOLLOWER
I20260810 07:25:06.121790 16571 raft_consensus.cc:348] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } } peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } }
I20260810 07:25:06.122035 16571 raft_consensus.cc:374] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:06.122095 16571 raft_consensus.cc:729] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2c0e0687ac5456584fc4db0949c2f12, State: Initialized, Role: FOLLOWER
I20260810 07:25:06.122192 16573 raft_consensus.cc:348] T a0c2f91618ed4e88855c1c76a3754502 P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } } peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } }
I20260810 07:25:06.122367 16568 consensus_queue.cc:260] T a0c2f91618ed4e88855c1c76a3754502 P cfc9dd8eb7bb4fcc9068037c315efa1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } } peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } }
I20260810 07:25:06.122480 16573 raft_consensus.cc:374] T a0c2f91618ed4e88855c1c76a3754502 P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:06.122457 16571 consensus_queue.cc:260] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } } peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } }
I20260810 07:25:06.122577 16573 raft_consensus.cc:729] T a0c2f91618ed4e88855c1c76a3754502 P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c8cbc8254d40fba61eaa0984a3fe47, State: Initialized, Role: FOLLOWER
I20260810 07:25:06.123087 16573 consensus_queue.cc:260] T a0c2f91618ed4e88855c1c76a3754502 P f7c8cbc8254d40fba61eaa0984a3fe47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } } peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } }
I20260810 07:25:06.123230 16100 heartbeater.cc:503] Master 127.12.157.254:36711 was elected leader, sending a full tablet report...
I20260810 07:25:06.123471 16571 ts_tablet_manager.cc:1428] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 07:25:06.123489 16568 ts_tablet_manager.cc:1428] T a0c2f91618ed4e88855c1c76a3754502 P cfc9dd8eb7bb4fcc9068037c315efa1e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 07:25:06.123814 16319 heartbeater.cc:503] Master 127.12.157.254:36711 was elected leader, sending a full tablet report...
I20260810 07:25:06.125167 16530 heartbeater.cc:503] Master 127.12.157.254:36711 was elected leader, sending a full tablet report...
I20260810 07:25:06.125461 16573 ts_tablet_manager.cc:1428] T a0c2f91618ed4e88855c1c76a3754502 P f7c8cbc8254d40fba61eaa0984a3fe47: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 07:25:06.137842 16582 raft_consensus.cc:482] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:25:06.138123 16582 raft_consensus.cc:504] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } } peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } }
I20260810 07:25:06.139480 16582 leader_election.cc:288] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7c8cbc8254d40fba61eaa0984a3fe47 (127.12.157.195:34729), cfc9dd8eb7bb4fcc9068037c315efa1e (127.12.157.193:41553)
I20260810 07:25:06.144105 16157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.139676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:34729 (local address 127.12.157.194:40251)
0810 07:25:06.139789 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:06.139807 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:06.139902 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:06.140326 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:06.140329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:06.140350 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:06.140492 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.140501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.141488 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.141491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.142695 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.142705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.143664 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.143667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.143758 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.143801 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 07:25:06.143866 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 07:25:06.143926 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 07:25:06.144146 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.139800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:40251 (local address 127.12.157.195:34729)
0810 07:25:06.139938 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:06.139942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:06.139961 (+ 19us) server_negotiation.cc:407] Connection header received
0810 07:25:06.140019 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:06.140024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:06.140098 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:06.140195 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:06.140636 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.141349 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.142875 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.143533 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.143664 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.143718 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:06.143913 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 07:25:06.144023 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 07:25:06.144567 16564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.140174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:34085 (local address 127.12.157.193:41553)
0810 07:25:06.140306 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:06.140310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:06.140467 (+ 157us) server_negotiation.cc:407] Connection header received
0810 07:25:06.140579 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:06.140585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:06.140640 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:06.140719 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:06.141334 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.141867 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.143137 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.143927 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.144049 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.144095 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:06.144350 (+ 255us) server_negotiation.cc:299] Negotiation successful
0810 07:25:06.144432 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 07:25:06.144817 16464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0c2f91618ed4e88855c1c76a3754502" candidate_uuid: "d2c0e0687ac5456584fc4db0949c2f12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" is_pre_election: true
I20260810 07:25:06.144924 16584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.139960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:41553 (local address 127.12.157.194:34085)
0810 07:25:06.140338 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:06.140351 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:06.140459 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:06.140728 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:06.140730 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:06.140746 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:06.141201 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.141210 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.141976 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.141980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.143021 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.143029 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.144055 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.144059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.144165 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.144196 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 07:25:06.144709 (+ 513us) client_negotiation.cc:240] Negotiation successful
0810 07:25:06.144783 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":169,"threads_started":1}
I20260810 07:25:06.145197 16464 raft_consensus.cc:2454] T a0c2f91618ed4e88855c1c76a3754502 P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2c0e0687ac5456584fc4db0949c2f12 in term 0.
I20260810 07:25:06.145707 16143 leader_election.cc:302] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2c0e0687ac5456584fc4db0949c2f12, f7c8cbc8254d40fba61eaa0984a3fe47; no voters:
I20260810 07:25:06.145771 16034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0c2f91618ed4e88855c1c76a3754502" candidate_uuid: "d2c0e0687ac5456584fc4db0949c2f12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" is_pre_election: true
I20260810 07:25:06.146027 16034 raft_consensus.cc:2454] T a0c2f91618ed4e88855c1c76a3754502 P cfc9dd8eb7bb4fcc9068037c315efa1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2c0e0687ac5456584fc4db0949c2f12 in term 0.
I20260810 07:25:06.146028 16582 raft_consensus.cc:2781] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:25:06.146111 16582 raft_consensus.cc:482] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:25:06.146155 16582 raft_consensus.cc:3037] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:06.147349 16582 raft_consensus.cc:504] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } } peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } }
I20260810 07:25:06.147723 16582 leader_election.cc:288] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12 [CANDIDATE]: Term 1 election: Requested vote from peers f7c8cbc8254d40fba61eaa0984a3fe47 (127.12.157.195:34729), cfc9dd8eb7bb4fcc9068037c315efa1e (127.12.157.193:41553)
I20260810 07:25:06.148092 16464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0c2f91618ed4e88855c1c76a3754502" candidate_uuid: "d2c0e0687ac5456584fc4db0949c2f12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47"
I20260810 07:25:06.148221 16034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0c2f91618ed4e88855c1c76a3754502" candidate_uuid: "d2c0e0687ac5456584fc4db0949c2f12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e"
I20260810 07:25:06.148218 16464 raft_consensus.cc:3037] T a0c2f91618ed4e88855c1c76a3754502 P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:06.148381 16034 raft_consensus.cc:3037] T a0c2f91618ed4e88855c1c76a3754502 P cfc9dd8eb7bb4fcc9068037c315efa1e [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:06.149423 16464 raft_consensus.cc:2454] T a0c2f91618ed4e88855c1c76a3754502 P f7c8cbc8254d40fba61eaa0984a3fe47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2c0e0687ac5456584fc4db0949c2f12 in term 1.
I20260810 07:25:06.149643 16034 raft_consensus.cc:2454] T a0c2f91618ed4e88855c1c76a3754502 P cfc9dd8eb7bb4fcc9068037c315efa1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2c0e0687ac5456584fc4db0949c2f12 in term 1.
I20260810 07:25:06.149736 16143 leader_election.cc:302] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2c0e0687ac5456584fc4db0949c2f12, f7c8cbc8254d40fba61eaa0984a3fe47; no voters:
I20260810 07:25:06.149901 16582 raft_consensus.cc:2781] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:06.150151 16582 raft_consensus.cc:686] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12 [term 1 LEADER]: Becoming Leader. State: Replica: d2c0e0687ac5456584fc4db0949c2f12, State: Running, Role: LEADER
I20260810 07:25:06.150648 16582 consensus_queue.cc:237] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } } peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } }
I20260810 07:25:06.154253 15762 catalog_manager.cc:5495] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2c0e0687ac5456584fc4db0949c2f12 (127.12.157.194). New cstate: current_term: 1 leader_uuid: "d2c0e0687ac5456584fc4db0949c2f12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } health_report { overall_health: HEALTHY } } }
Aug 10 07:25:06 dist-test-slave-ffrd krb5kdc[15699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346705, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:06.199995 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.191103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:36711 (local address 127.0.0.1:42566)
0810 07:25:06.191438 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:06.191451 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:06.191600 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:06.191862 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:06.191865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:06.192292 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:06.192388 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.192395 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.193260 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.193263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.193826 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.193834 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.194280 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.194284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.194382 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.195000 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:06.195013 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:06.196683 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:06.198421 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:06.198431 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:06.198436 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:06.198691 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:06.198972 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:06.198974 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:06.198975 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:06.199056 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:06.199455 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:06.199461 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:06.199582 (+ 121us) client_negotiation.cc:769] Sending connection context
0810 07:25:06.199690 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 07:25:06.199837 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":138,"threads_started":1}
I20260810 07:25:06.200021 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.191138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42566 (local address 127.12.157.254:36711)
0810 07:25:06.191295 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:06.191302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:06.191535 (+ 233us) server_negotiation.cc:407] Connection header received
0810 07:25:06.191716 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:06.191719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:06.191760 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:06.191844 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:06.192512 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.193145 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.193951 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.194161 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.194267 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.196857 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:06.196860 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:06.196862 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:06.196886 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:06.198293 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:06.198787 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:06.198790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:06.198791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:06.198885 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:06.199144 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:06.199147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:06.199148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:06.199327 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:06.199404 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:06.199744 (+ 340us) server_negotiation.cc:299] Negotiation successful
0810 07:25:06.199910 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":52}
I20260810 07:25:06.204174 15762 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:42566:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 07:25:06.205166 15762 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:25:06.210723 16218 tablet_service.cc:1467] Processing CreateTablet for tablet 635a0e2a45a44a5b865adb556b382926 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1c182cc8c2864b38a9273aaa7c1a64a0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 07:25:06.210702 15999 tablet_service.cc:1467] Processing CreateTablet for tablet 635a0e2a45a44a5b865adb556b382926 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1c182cc8c2864b38a9273aaa7c1a64a0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 07:25:06.210909 16427 tablet_service.cc:1467] Processing CreateTablet for tablet 635a0e2a45a44a5b865adb556b382926 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1c182cc8c2864b38a9273aaa7c1a64a0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 07:25:06.211493 16427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 635a0e2a45a44a5b865adb556b382926. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:06.211493 15999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 635a0e2a45a44a5b865adb556b382926. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:06.211503 16218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 635a0e2a45a44a5b865adb556b382926. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:06.214406 16571 tablet_bootstrap.cc:492] T 635a0e2a45a44a5b865adb556b382926 P d2c0e0687ac5456584fc4db0949c2f12: Bootstrap starting.
I20260810 07:25:06.214578 16568 tablet_bootstrap.cc:492] T 635a0e2a45a44a5b865adb556b382926 P cfc9dd8eb7bb4fcc9068037c315efa1e: Bootstrap starting.
I20260810 07:25:06.214934 16573 tablet_bootstrap.cc:492] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47: Bootstrap starting.
I20260810 07:25:06.216008 16573 tablet_bootstrap.cc:654] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:06.216071 16571 tablet_bootstrap.cc:654] T 635a0e2a45a44a5b865adb556b382926 P d2c0e0687ac5456584fc4db0949c2f12: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:06.216595 16568 tablet_bootstrap.cc:654] T 635a0e2a45a44a5b865adb556b382926 P cfc9dd8eb7bb4fcc9068037c315efa1e: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:06.217422 16573 tablet_bootstrap.cc:492] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47: No bootstrap required, opened a new log
I20260810 07:25:06.217545 16573 ts_tablet_manager.cc:1397] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 07:25:06.218022 16573 raft_consensus.cc:348] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:06.218146 16573 raft_consensus.cc:374] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:06.218194 16573 raft_consensus.cc:729] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c8cbc8254d40fba61eaa0984a3fe47, State: Initialized, Role: FOLLOWER
I20260810 07:25:06.218398 16568 tablet_bootstrap.cc:492] T 635a0e2a45a44a5b865adb556b382926 P cfc9dd8eb7bb4fcc9068037c315efa1e: No bootstrap required, opened a new log
I20260810 07:25:06.218384 16573 consensus_queue.cc:260] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:06.218505 16568 ts_tablet_manager.cc:1397] T 635a0e2a45a44a5b865adb556b382926 P cfc9dd8eb7bb4fcc9068037c315efa1e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260810 07:25:06.218679 16573 ts_tablet_manager.cc:1428] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 07:25:06.219012 16583 tablet_replica.cc:439] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } } }
I20260810 07:25:06.219085 16568 raft_consensus.cc:348] T 635a0e2a45a44a5b865adb556b382926 P cfc9dd8eb7bb4fcc9068037c315efa1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:06.219213 16583 tablet_replica.cc:442] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:25:06.219203 16568 raft_consensus.cc:374] T 635a0e2a45a44a5b865adb556b382926 P cfc9dd8eb7bb4fcc9068037c315efa1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:06.219252 16568 raft_consensus.cc:729] T 635a0e2a45a44a5b865adb556b382926 P cfc9dd8eb7bb4fcc9068037c315efa1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc9dd8eb7bb4fcc9068037c315efa1e, State: Initialized, Role: FOLLOWER
I20260810 07:25:06.219409 16568 consensus_queue.cc:260] T 635a0e2a45a44a5b865adb556b382926 P cfc9dd8eb7bb4fcc9068037c315efa1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:06.219687 16571 tablet_bootstrap.cc:492] T 635a0e2a45a44a5b865adb556b382926 P d2c0e0687ac5456584fc4db0949c2f12: No bootstrap required, opened a new log
I20260810 07:25:06.219686 16568 ts_tablet_manager.cc:1428] T 635a0e2a45a44a5b865adb556b382926 P cfc9dd8eb7bb4fcc9068037c315efa1e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 07:25:06.219786 16571 ts_tablet_manager.cc:1397] T 635a0e2a45a44a5b865adb556b382926 P d2c0e0687ac5456584fc4db0949c2f12: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260810 07:25:06.219717 16581 tablet_replica.cc:439] T 635a0e2a45a44a5b865adb556b382926 P cfc9dd8eb7bb4fcc9068037c315efa1e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } } }
I20260810 07:25:06.219873 16581 tablet_replica.cc:442] T 635a0e2a45a44a5b865adb556b382926 P cfc9dd8eb7bb4fcc9068037c315efa1e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:25:06.220273 16571 raft_consensus.cc:348] T 635a0e2a45a44a5b865adb556b382926 P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:06.220420 16571 raft_consensus.cc:374] T 635a0e2a45a44a5b865adb556b382926 P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:06.220463 16571 raft_consensus.cc:729] T 635a0e2a45a44a5b865adb556b382926 P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2c0e0687ac5456584fc4db0949c2f12, State: Initialized, Role: FOLLOWER
I20260810 07:25:06.220592 16571 consensus_queue.cc:260] T 635a0e2a45a44a5b865adb556b382926 P d2c0e0687ac5456584fc4db0949c2f12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:06.220891 16571 ts_tablet_manager.cc:1428] T 635a0e2a45a44a5b865adb556b382926 P d2c0e0687ac5456584fc4db0949c2f12: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 07:25:06.221123 16582 tablet_replica.cc:439] T 635a0e2a45a44a5b865adb556b382926 P d2c0e0687ac5456584fc4db0949c2f12: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } } }
I20260810 07:25:06.221277 16582 tablet_replica.cc:442] T 635a0e2a45a44a5b865adb556b382926 P d2c0e0687ac5456584fc4db0949c2f12: This TxnStatusTablet replica's current role is: FOLLOWER
W20260810 07:25:06.285571 16531 tablet.cc:2369] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 07:25:06.301095 16323 tablet.cc:2369] T 635a0e2a45a44a5b865adb556b382926 P d2c0e0687ac5456584fc4db0949c2f12: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 07:25:06.303165 16102 tablet.cc:2369] T 635a0e2a45a44a5b865adb556b382926 P cfc9dd8eb7bb4fcc9068037c315efa1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:25:06.307878 16583 raft_consensus.cc:482] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:25:06.308018 16583 raft_consensus.cc:504] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:06.309367 16583 leader_election.cc:288] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfc9dd8eb7bb4fcc9068037c315efa1e (127.12.157.193:41553), d2c0e0687ac5456584fc4db0949c2f12 (127.12.157.194:46585)
I20260810 07:25:06.314626 16370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.309672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:41553 (local address 127.12.157.195:36303)
0810 07:25:06.309824 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:06.309842 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:06.309976 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:06.310319 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:06.310323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:06.310358 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:06.310497 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.310505 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.311590 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.311593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.313157 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.313170 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.314133 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.314140 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.314275 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.314304 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 07:25:06.314386 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 07:25:06.314464 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":66}
I20260810 07:25:06.314651 16564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.309819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:36303 (local address 127.12.157.193:41553)
0810 07:25:06.309966 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:06.309970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:06.309986 (+ 16us) server_negotiation.cc:407] Connection header received
0810 07:25:06.310131 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:06.310134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:06.310193 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:06.310297 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:06.310639 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.311422 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.313310 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.313988 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.314123 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.314189 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:06.314425 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 07:25:06.314497 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 07:25:06.314780 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.310121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:46585 (local address 127.12.157.195:40007)
0810 07:25:06.310601 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:06.310617 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:06.310710 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:06.310959 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:06.310961 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:06.310976 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:06.311071 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.311076 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.311921 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.311924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.313379 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.313396 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.314346 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.314352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.314445 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.314464 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 07:25:06.314536 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 07:25:06.314590 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":81,"thread_start_us":182,"threads_started":1}
I20260810 07:25:06.314802 16567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.310302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:40007 (local address 127.12.157.194:46585)
0810 07:25:06.310444 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:06.310449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:06.310709 (+ 260us) server_negotiation.cc:407] Connection header received
0810 07:25:06.310802 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:06.310805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:06.310854 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:06.310943 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:06.311186 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.311799 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.313518 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.314082 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.314204 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.314262 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:06.314575 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 07:25:06.314653 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 07:25:06.315444 16034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "635a0e2a45a44a5b865adb556b382926" candidate_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" is_pre_election: true
I20260810 07:25:06.315531 16247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "635a0e2a45a44a5b865adb556b382926" candidate_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2c0e0687ac5456584fc4db0949c2f12" is_pre_election: true
I20260810 07:25:06.315631 16034 raft_consensus.cc:2454] T 635a0e2a45a44a5b865adb556b382926 P cfc9dd8eb7bb4fcc9068037c315efa1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c8cbc8254d40fba61eaa0984a3fe47 in term 0.
I20260810 07:25:06.315831 16247 raft_consensus.cc:2454] T 635a0e2a45a44a5b865adb556b382926 P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c8cbc8254d40fba61eaa0984a3fe47 in term 0.
I20260810 07:25:06.316103 16358 leader_election.cc:302] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfc9dd8eb7bb4fcc9068037c315efa1e, f7c8cbc8254d40fba61eaa0984a3fe47; no voters:
I20260810 07:25:06.316428 16583 raft_consensus.cc:2781] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:25:06.316493 16583 raft_consensus.cc:482] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:25:06.316521 16583 raft_consensus.cc:3037] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:06.317646 16583 raft_consensus.cc:504] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:06.318058 16583 leader_election.cc:288] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47 [CANDIDATE]: Term 1 election: Requested vote from peers cfc9dd8eb7bb4fcc9068037c315efa1e (127.12.157.193:41553), d2c0e0687ac5456584fc4db0949c2f12 (127.12.157.194:46585)
I20260810 07:25:06.318427 16034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "635a0e2a45a44a5b865adb556b382926" candidate_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e"
I20260810 07:25:06.318495 16247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "635a0e2a45a44a5b865adb556b382926" candidate_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2c0e0687ac5456584fc4db0949c2f12"
I20260810 07:25:06.318574 16034 raft_consensus.cc:3037] T 635a0e2a45a44a5b865adb556b382926 P cfc9dd8eb7bb4fcc9068037c315efa1e [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:06.318605 16247 raft_consensus.cc:3037] T 635a0e2a45a44a5b865adb556b382926 P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:06.319782 16247 raft_consensus.cc:2454] T 635a0e2a45a44a5b865adb556b382926 P d2c0e0687ac5456584fc4db0949c2f12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c8cbc8254d40fba61eaa0984a3fe47 in term 1.
I20260810 07:25:06.319782 16034 raft_consensus.cc:2454] T 635a0e2a45a44a5b865adb556b382926 P cfc9dd8eb7bb4fcc9068037c315efa1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c8cbc8254d40fba61eaa0984a3fe47 in term 1.
I20260810 07:25:06.320078 16358 leader_election.cc:302] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfc9dd8eb7bb4fcc9068037c315efa1e, f7c8cbc8254d40fba61eaa0984a3fe47; no voters:
I20260810 07:25:06.320271 16583 raft_consensus.cc:2781] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:06.320545 16583 raft_consensus.cc:686] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47 [term 1 LEADER]: Becoming Leader. State: Replica: f7c8cbc8254d40fba61eaa0984a3fe47, State: Running, Role: LEADER
I20260810 07:25:06.321036 16583 consensus_queue.cc:237] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:06.323632 16607 tablet_replica.cc:439] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47: TxnStatusTablet state changed. Reason: New leader f7c8cbc8254d40fba61eaa0984a3fe47. Latest consensus state: current_term: 1 leader_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } } }
I20260810 07:25:06.323764 16607 tablet_replica.cc:442] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47: This TxnStatusTablet replica's current role is: LEADER
I20260810 07:25:06.324100 16612 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 07:25:06.324364 16612 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 07:25:06.324982 15766 catalog_manager.cc:5495] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7c8cbc8254d40fba61eaa0984a3fe47 (127.12.157.195). New cstate: current_term: 1 leader_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } health_report { overall_health: HEALTHY } } }
I20260810 07:25:06.349858 16599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.345366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:34729 (local address 127.0.0.1:37724)
0810 07:25:06.345542 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:06.345561 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:06.345664 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:06.346078 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:06.346080 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:06.346088 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 07:25:06.346188 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.346193 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.347231 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.347234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.347853 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.347862 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.348491 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.348494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.348584 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.348632 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 07:25:06.349523 (+ 891us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 07:25:06.349541 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 07:25:06.349621 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 07:25:06.349711 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 07:25:06.349902 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.345482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37724 (local address 127.12.157.195:34729)
0810 07:25:06.345615 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:06.345619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:06.345635 (+ 16us) server_negotiation.cc:407] Connection header received
0810 07:25:06.345855 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:06.345859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:06.345929 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:06.346039 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:25:06.346309 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.347104 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.347978 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.348334 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.348451 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.348799 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:25:06.349395 (+ 596us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:25:06.349512 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:06.349658 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 07:25:06.349729 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 07:25:06.355356 16034 raft_consensus.cc:1261] T 635a0e2a45a44a5b865adb556b382926 P cfc9dd8eb7bb4fcc9068037c315efa1e [term 1 FOLLOWER]: Refusing update from remote peer f7c8cbc8254d40fba61eaa0984a3fe47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:06.355360 16247 raft_consensus.cc:1261] T 635a0e2a45a44a5b865adb556b382926 P d2c0e0687ac5456584fc4db0949c2f12 [term 1 FOLLOWER]: Refusing update from remote peer f7c8cbc8254d40fba61eaa0984a3fe47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:06.356118 16607 consensus_queue.cc:1035] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:06.356344 16583 consensus_queue.cc:1035] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:06.359678 16582 tablet_replica.cc:439] T 635a0e2a45a44a5b865adb556b382926 P d2c0e0687ac5456584fc4db0949c2f12: TxnStatusTablet state changed. Reason: New leader f7c8cbc8254d40fba61eaa0984a3fe47. Latest consensus state: current_term: 1 leader_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } } }
I20260810 07:25:06.359812 16582 tablet_replica.cc:442] T 635a0e2a45a44a5b865adb556b382926 P d2c0e0687ac5456584fc4db0949c2f12: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:25:06.360102 16581 tablet_replica.cc:439] T 635a0e2a45a44a5b865adb556b382926 P cfc9dd8eb7bb4fcc9068037c315efa1e: TxnStatusTablet state changed. Reason: New leader f7c8cbc8254d40fba61eaa0984a3fe47. Latest consensus state: current_term: 1 leader_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } } }
I20260810 07:25:06.360212 16581 tablet_replica.cc:442] T 635a0e2a45a44a5b865adb556b382926 P cfc9dd8eb7bb4fcc9068037c315efa1e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:25:06.364229 16583 tablet_replica.cc:439] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } } }
I20260810 07:25:06.364352 16583 tablet_replica.cc:442] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47: This TxnStatusTablet replica's current role is: LEADER
I20260810 07:25:06.364868 16581 tablet_replica.cc:439] T 635a0e2a45a44a5b865adb556b382926 P cfc9dd8eb7bb4fcc9068037c315efa1e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } } }
I20260810 07:25:06.365012 16581 tablet_replica.cc:442] T 635a0e2a45a44a5b865adb556b382926 P cfc9dd8eb7bb4fcc9068037c315efa1e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:25:06.365743 16627 mvcc.cc:205] Tried to move back new op lower bound from 7316876109224574976 to 7316876109094109184. Current Snapshot: MvccSnapshot[applied={T|T < 7316876109224574976}]
I20260810 07:25:06.374519 16607 tablet_replica.cc:439] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } } }
I20260810 07:25:06.374665 16607 tablet_replica.cc:442] T 635a0e2a45a44a5b865adb556b382926 P f7c8cbc8254d40fba61eaa0984a3fe47: This TxnStatusTablet replica's current role is: LEADER
I20260810 07:25:06.375079 16567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.370552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58638 (local address 127.12.157.194:46585)
0810 07:25:06.370679 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:06.370683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:06.370700 (+ 17us) server_negotiation.cc:407] Connection header received
0810 07:25:06.370814 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:06.370817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:06.370880 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:06.370992 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:25:06.371294 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.372088 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.373046 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.373394 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.373504 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.373836 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:25:06.374343 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:25:06.374449 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:06.374887 (+ 438us) server_negotiation.cc:299] Negotiation successful
0810 07:25:06.374943 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 07:25:06.375854 16582 tablet_replica.cc:439] T 635a0e2a45a44a5b865adb556b382926 P d2c0e0687ac5456584fc4db0949c2f12: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } } }
I20260810 07:25:06.375993 16582 tablet_replica.cc:442] T 635a0e2a45a44a5b865adb556b382926 P d2c0e0687ac5456584fc4db0949c2f12: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:25:06.376734 16626 mvcc.cc:205] Tried to move back new op lower bound from 7316876109224574976 to 7316876109094109184. Current Snapshot: MvccSnapshot[applied={T|T < 7316876109224574976}]
I20260810 07:25:06.389089 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.384478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:57965 (local address 127.12.157.195:34729)
0810 07:25:06.384613 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:06.384617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:06.384871 (+ 254us) server_negotiation.cc:407] Connection header received
0810 07:25:06.385053 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:06.385057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:06.385147 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:06.385269 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:25:06.385574 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.386462 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.387737 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.388083 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.388153 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.388586 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:25:06.388666 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:25:06.388708 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:06.388875 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 07:25:06.388930 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 07:25:06.389444 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.384308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:34729 (local address 127.12.157.194:57965)
0810 07:25:06.384752 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:06.384777 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:06.384881 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:06.385302 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:06.385305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:06.385313 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 07:25:06.385449 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.385455 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.386588 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.386591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.387598 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.387608 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.388262 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.388265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.388362 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.388412 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 07:25:06.388778 (+ 366us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 07:25:06.388783 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:25:06.389213 (+ 430us) client_negotiation.cc:240] Negotiation successful
0810 07:25:06.389293 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":136,"threads_started":1}
I20260810 07:25:06.402491 16034 raft_consensus.cc:1261] T a0c2f91618ed4e88855c1c76a3754502 P cfc9dd8eb7bb4fcc9068037c315efa1e [term 1 FOLLOWER]: Refusing update from remote peer d2c0e0687ac5456584fc4db0949c2f12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:06.402734 16464 raft_consensus.cc:1261] T a0c2f91618ed4e88855c1c76a3754502 P f7c8cbc8254d40fba61eaa0984a3fe47 [term 1 FOLLOWER]: Refusing update from remote peer d2c0e0687ac5456584fc4db0949c2f12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:06.402988 16586 consensus_queue.cc:1035] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:06.403446 16586 consensus_queue.cc:1035] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:06.412425 16624 mvcc.cc:205] Tried to move back new op lower bound from 7316876109409120256 to 7316876108396584960. Current Snapshot: MvccSnapshot[applied={T|T < 7316876109409120256}]
I20260810 07:25:06.433171 16567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.429109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:56935 (local address 127.12.157.194:46585)
0810 07:25:06.429218 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:06.429223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:06.429258 (+ 35us) server_negotiation.cc:407] Connection header received
0810 07:25:06.429303 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:06.429305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:06.429363 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:06.429441 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:25:06.429775 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.430336 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.431547 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.431971 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.432102 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.432546 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:25:06.432635 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:25:06.432701 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:06.432916 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 07:25:06.433019 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 07:25:06.433758 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.428870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:46585 (local address 127.12.157.195:56935)
0810 07:25:06.429065 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:06.429082 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:06.429180 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:06.429501 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:06.429504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:06.429513 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 07:25:06.429648 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.429655 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.430439 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.430443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.431418 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.431430 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.432127 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.432134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.432291 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.432350 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 07:25:06.432765 (+ 415us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 07:25:06.432772 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 07:25:06.432833 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 07:25:06.433626 (+ 793us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 07:25:06.461606 15766 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:06.461804 15766 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:06.464295 15766 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f64d8b2626d44b29f277990aa61bf4e: Sending DeleteTablet for 3 replicas of tablet a0c2f91618ed4e88855c1c76a3754502
I20260810 07:25:06.465065 15999 tablet_service.cc:1514] Processing DeleteTablet for tablet a0c2f91618ed4e88855c1c76a3754502 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:06 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:36656
I20260810 07:25:06.465113 16218 tablet_service.cc:1514] Processing DeleteTablet for tablet a0c2f91618ed4e88855c1c76a3754502 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:06 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:58630
I20260810 07:25:06.465684 16427 tablet_service.cc:1514] Processing DeleteTablet for tablet a0c2f91618ed4e88855c1c76a3754502 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:06 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:37718
I20260810 07:25:06.466149 16656 tablet_replica.cc:330] T a0c2f91618ed4e88855c1c76a3754502 P f7c8cbc8254d40fba61eaa0984a3fe47: stopping tablet replica
I20260810 07:25:06.466415 15766 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 07:25:06.466799 16656 raft_consensus.cc:2229] T a0c2f91618ed4e88855c1c76a3754502 P f7c8cbc8254d40fba61eaa0984a3fe47 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260810 07:25:06.466876 15766 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:25:06.467104 16656 raft_consensus.cc:2258] T a0c2f91618ed4e88855c1c76a3754502 P f7c8cbc8254d40fba61eaa0984a3fe47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:06.468855 16656 ts_tablet_manager.cc:1905] T a0c2f91618ed4e88855c1c76a3754502 P f7c8cbc8254d40fba61eaa0984a3fe47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:06.472429 16656 ts_tablet_manager.cc:1918] T a0c2f91618ed4e88855c1c76a3754502 P f7c8cbc8254d40fba61eaa0984a3fe47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 07:25:06.472610 16656 log.cc:1196] T a0c2f91618ed4e88855c1c76a3754502 P f7c8cbc8254d40fba61eaa0984a3fe47: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-2/wal/wals/a0c2f91618ed4e88855c1c76a3754502
I20260810 07:25:06.472591 15999 tablet_service.cc:1467] Processing CreateTablet for tablet 18ddefc5dadd4c8e98011f73baa18ecb (DEFAULT_TABLE table=test-table [id=363505947b0046e1b44c6c7bb93b050d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:06.472982 15999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18ddefc5dadd4c8e98011f73baa18ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:06.473210 16656 ts_tablet_manager.cc:1939] T a0c2f91618ed4e88855c1c76a3754502 P f7c8cbc8254d40fba61eaa0984a3fe47: Deleting consensus metadata
I20260810 07:25:06.475298 16568 tablet_bootstrap.cc:492] T 18ddefc5dadd4c8e98011f73baa18ecb P cfc9dd8eb7bb4fcc9068037c315efa1e: Bootstrap starting.
I20260810 07:25:06.475648 15748 catalog_manager.cc:4841] TS f7c8cbc8254d40fba61eaa0984a3fe47 (127.12.157.195:34729): tablet a0c2f91618ed4e88855c1c76a3754502 (table test-table [id=3a5d12521bce4f58a9c0d0b3b63bedf3]) successfully deleted
I20260810 07:25:06.476310 16568 tablet_bootstrap.cc:654] T 18ddefc5dadd4c8e98011f73baa18ecb P cfc9dd8eb7bb4fcc9068037c315efa1e: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:06.476248 16218 tablet_service.cc:1467] Processing CreateTablet for tablet 18ddefc5dadd4c8e98011f73baa18ecb (DEFAULT_TABLE table=test-table [id=363505947b0046e1b44c6c7bb93b050d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:06.476543 16218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18ddefc5dadd4c8e98011f73baa18ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:06.477730 16568 tablet_bootstrap.cc:492] T 18ddefc5dadd4c8e98011f73baa18ecb P cfc9dd8eb7bb4fcc9068037c315efa1e: No bootstrap required, opened a new log
I20260810 07:25:06.477816 16568 ts_tablet_manager.cc:1397] T 18ddefc5dadd4c8e98011f73baa18ecb P cfc9dd8eb7bb4fcc9068037c315efa1e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260810 07:25:06.478288 16568 raft_consensus.cc:348] T 18ddefc5dadd4c8e98011f73baa18ecb P cfc9dd8eb7bb4fcc9068037c315efa1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:06.478410 16568 raft_consensus.cc:374] T 18ddefc5dadd4c8e98011f73baa18ecb P cfc9dd8eb7bb4fcc9068037c315efa1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:06.478461 16568 raft_consensus.cc:729] T 18ddefc5dadd4c8e98011f73baa18ecb P cfc9dd8eb7bb4fcc9068037c315efa1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc9dd8eb7bb4fcc9068037c315efa1e, State: Initialized, Role: FOLLOWER
I20260810 07:25:06.478582 16568 consensus_queue.cc:260] T 18ddefc5dadd4c8e98011f73baa18ecb P cfc9dd8eb7bb4fcc9068037c315efa1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:06.478834 16568 ts_tablet_manager.cc:1428] T 18ddefc5dadd4c8e98011f73baa18ecb P cfc9dd8eb7bb4fcc9068037c315efa1e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 07:25:06.479527 16427 tablet_service.cc:1467] Processing CreateTablet for tablet 18ddefc5dadd4c8e98011f73baa18ecb (DEFAULT_TABLE table=test-table [id=363505947b0046e1b44c6c7bb93b050d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:06.479804 16427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18ddefc5dadd4c8e98011f73baa18ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:06.482136 16573 tablet_bootstrap.cc:492] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47: Bootstrap starting.
I20260810 07:25:06.483174 16573 tablet_bootstrap.cc:654] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:06.483793 16571 tablet_bootstrap.cc:492] T 18ddefc5dadd4c8e98011f73baa18ecb P d2c0e0687ac5456584fc4db0949c2f12: Bootstrap starting.
I20260810 07:25:06.484560 16573 tablet_bootstrap.cc:492] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47: No bootstrap required, opened a new log
I20260810 07:25:06.484649 16573 ts_tablet_manager.cc:1397] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 07:25:06.484877 16571 tablet_bootstrap.cc:654] T 18ddefc5dadd4c8e98011f73baa18ecb P d2c0e0687ac5456584fc4db0949c2f12: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:06.485216 16573 raft_consensus.cc:348] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:06.485330 16573 raft_consensus.cc:374] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:06.485363 16573 raft_consensus.cc:729] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c8cbc8254d40fba61eaa0984a3fe47, State: Initialized, Role: FOLLOWER
I20260810 07:25:06.485522 16573 consensus_queue.cc:260] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:06.485814 16573 ts_tablet_manager.cc:1428] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:06.487534 16571 tablet_bootstrap.cc:492] T 18ddefc5dadd4c8e98011f73baa18ecb P d2c0e0687ac5456584fc4db0949c2f12: No bootstrap required, opened a new log
I20260810 07:25:06.487629 16571 ts_tablet_manager.cc:1397] T 18ddefc5dadd4c8e98011f73baa18ecb P d2c0e0687ac5456584fc4db0949c2f12: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260810 07:25:06.488134 16571 raft_consensus.cc:348] T 18ddefc5dadd4c8e98011f73baa18ecb P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:06.488248 16571 raft_consensus.cc:374] T 18ddefc5dadd4c8e98011f73baa18ecb P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:06.488283 16571 raft_consensus.cc:729] T 18ddefc5dadd4c8e98011f73baa18ecb P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2c0e0687ac5456584fc4db0949c2f12, State: Initialized, Role: FOLLOWER
I20260810 07:25:06.488560 16571 consensus_queue.cc:260] T 18ddefc5dadd4c8e98011f73baa18ecb P d2c0e0687ac5456584fc4db0949c2f12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:06.489450 16571 ts_tablet_manager.cc:1428] T 18ddefc5dadd4c8e98011f73baa18ecb P d2c0e0687ac5456584fc4db0949c2f12: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260810 07:25:06.490396 16653 tablet_replica.cc:330] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12: stopping tablet replica
I20260810 07:25:06.490749 16653 raft_consensus.cc:2229] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:25:06.492338 16653 raft_consensus.cc:2258] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:06.492810 16654 tablet_replica.cc:330] T a0c2f91618ed4e88855c1c76a3754502 P cfc9dd8eb7bb4fcc9068037c315efa1e: stopping tablet replica
I20260810 07:25:06.493213 16654 raft_consensus.cc:2229] T a0c2f91618ed4e88855c1c76a3754502 P cfc9dd8eb7bb4fcc9068037c315efa1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:25:06.493479 16654 raft_consensus.cc:2258] T a0c2f91618ed4e88855c1c76a3754502 P cfc9dd8eb7bb4fcc9068037c315efa1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:06.495178 16653 ts_tablet_manager.cc:1905] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:06.499167 16653 ts_tablet_manager.cc:1918] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 07:25:06.499272 16653 log.cc:1196] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-1/wal/wals/a0c2f91618ed4e88855c1c76a3754502
I20260810 07:25:06.499639 16653 ts_tablet_manager.cc:1939] T a0c2f91618ed4e88855c1c76a3754502 P d2c0e0687ac5456584fc4db0949c2f12: Deleting consensus metadata
I20260810 07:25:06.501997 15748 catalog_manager.cc:4841] TS d2c0e0687ac5456584fc4db0949c2f12 (127.12.157.194:46585): tablet a0c2f91618ed4e88855c1c76a3754502 (table test-table [id=3a5d12521bce4f58a9c0d0b3b63bedf3]) successfully deleted
I20260810 07:25:06.504341 16654 ts_tablet_manager.cc:1905] T a0c2f91618ed4e88855c1c76a3754502 P cfc9dd8eb7bb4fcc9068037c315efa1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:06.507241 16654 ts_tablet_manager.cc:1918] T a0c2f91618ed4e88855c1c76a3754502 P cfc9dd8eb7bb4fcc9068037c315efa1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 07:25:06.507366 16654 log.cc:1196] T a0c2f91618ed4e88855c1c76a3754502 P cfc9dd8eb7bb4fcc9068037c315efa1e: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-0/wal/wals/a0c2f91618ed4e88855c1c76a3754502
I20260810 07:25:06.507661 16654 ts_tablet_manager.cc:1939] T a0c2f91618ed4e88855c1c76a3754502 P cfc9dd8eb7bb4fcc9068037c315efa1e: Deleting consensus metadata
I20260810 07:25:06.508922 15748 catalog_manager.cc:4841] TS cfc9dd8eb7bb4fcc9068037c315efa1e (127.12.157.193:41553): tablet a0c2f91618ed4e88855c1c76a3754502 (table test-table [id=3a5d12521bce4f58a9c0d0b3b63bedf3]) successfully deleted
I20260810 07:25:06.519524 16636 raft_consensus.cc:482] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:25:06.519814 16636 raft_consensus.cc:504] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:06.520714 16636 leader_election.cc:288] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfc9dd8eb7bb4fcc9068037c315efa1e (127.12.157.193:41553), d2c0e0687ac5456584fc4db0949c2f12 (127.12.157.194:46585)
I20260810 07:25:06.521272 16032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18ddefc5dadd4c8e98011f73baa18ecb" candidate_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" is_pre_election: true
I20260810 07:25:06.521446 16032 raft_consensus.cc:2454] T 18ddefc5dadd4c8e98011f73baa18ecb P cfc9dd8eb7bb4fcc9068037c315efa1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c8cbc8254d40fba61eaa0984a3fe47 in term 0.
I20260810 07:25:06.521752 16247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18ddefc5dadd4c8e98011f73baa18ecb" candidate_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2c0e0687ac5456584fc4db0949c2f12" is_pre_election: true
I20260810 07:25:06.521919 16247 raft_consensus.cc:2454] T 18ddefc5dadd4c8e98011f73baa18ecb P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c8cbc8254d40fba61eaa0984a3fe47 in term 0.
I20260810 07:25:06.522262 16358 leader_election.cc:302] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfc9dd8eb7bb4fcc9068037c315efa1e, f7c8cbc8254d40fba61eaa0984a3fe47; no voters:
I20260810 07:25:06.522452 16636 raft_consensus.cc:2781] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:25:06.522570 16636 raft_consensus.cc:482] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:25:06.522632 16636 raft_consensus.cc:3037] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:06.523602 16636 raft_consensus.cc:504] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:06.523980 16636 leader_election.cc:288] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47 [CANDIDATE]: Term 1 election: Requested vote from peers cfc9dd8eb7bb4fcc9068037c315efa1e (127.12.157.193:41553), d2c0e0687ac5456584fc4db0949c2f12 (127.12.157.194:46585)
I20260810 07:25:06.524312 16032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18ddefc5dadd4c8e98011f73baa18ecb" candidate_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e"
I20260810 07:25:06.524449 16032 raft_consensus.cc:3037] T 18ddefc5dadd4c8e98011f73baa18ecb P cfc9dd8eb7bb4fcc9068037c315efa1e [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:06.525104 16247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18ddefc5dadd4c8e98011f73baa18ecb" candidate_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2c0e0687ac5456584fc4db0949c2f12"
I20260810 07:25:06.525231 16247 raft_consensus.cc:3037] T 18ddefc5dadd4c8e98011f73baa18ecb P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:06.525611 16032 raft_consensus.cc:2454] T 18ddefc5dadd4c8e98011f73baa18ecb P cfc9dd8eb7bb4fcc9068037c315efa1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c8cbc8254d40fba61eaa0984a3fe47 in term 1.
I20260810 07:25:06.525931 16358 leader_election.cc:302] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfc9dd8eb7bb4fcc9068037c315efa1e, f7c8cbc8254d40fba61eaa0984a3fe47; no voters:
I20260810 07:25:06.526090 16636 raft_consensus.cc:2781] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:06.526512 16247 raft_consensus.cc:2454] T 18ddefc5dadd4c8e98011f73baa18ecb P d2c0e0687ac5456584fc4db0949c2f12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c8cbc8254d40fba61eaa0984a3fe47 in term 1.
I20260810 07:25:06.526526 16636 raft_consensus.cc:686] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47 [term 1 LEADER]: Becoming Leader. State: Replica: f7c8cbc8254d40fba61eaa0984a3fe47, State: Running, Role: LEADER
I20260810 07:25:06.526682 16636 consensus_queue.cc:237] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:06.529307 15762 catalog_manager.cc:5495] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7c8cbc8254d40fba61eaa0984a3fe47 (127.12.157.195). New cstate: current_term: 1 leader_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } health_report { overall_health: HEALTHY } } }
I20260810 07:25:06.549484 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.546112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37728 (local address 127.12.157.195:34729)
0810 07:25:06.546226 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:06.546230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:06.546431 (+ 201us) server_negotiation.cc:407] Connection header received
0810 07:25:06.546626 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:06.546628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:06.546672 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:06.546764 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:25:06.547018 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.547549 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.548370 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.548608 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.548717 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.549012 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:25:06.549094 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:25:06.549178 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:06.549276 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 07:25:06.549327 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 07:25:06.561484 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.557457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:37589 (local address 127.12.157.195:34729)
0810 07:25:06.557566 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:06.557569 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:06.557581 (+ 12us) server_negotiation.cc:407] Connection header received
0810 07:25:06.557717 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:06.557722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:06.557771 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:06.557847 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:25:06.558173 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.559075 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.560256 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.560589 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.560689 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.561007 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:25:06.561080 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:25:06.561152 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:06.561253 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 07:25:06.561327 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":41}
I20260810 07:25:06.561497 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:06.557386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:34729 (local address 127.12.157.195:37589)
0810 07:25:06.557514 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:06.557529 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:06.557606 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:06.557906 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:06.557909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:06.557927 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 07:25:06.558050 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.558056 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.559186 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.559190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.560108 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:06.560137 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:06.560718 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:06.560722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:06.560816 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:06.560848 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 07:25:06.561171 (+ 323us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 07:25:06.561178 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 07:25:06.561247 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 07:25:06.561346 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":1}
I20260810 07:25:06.569002 16032 raft_consensus.cc:1261] T 18ddefc5dadd4c8e98011f73baa18ecb P cfc9dd8eb7bb4fcc9068037c315efa1e [term 1 FOLLOWER]: Refusing update from remote peer f7c8cbc8254d40fba61eaa0984a3fe47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:06.569042 16247 raft_consensus.cc:1261] T 18ddefc5dadd4c8e98011f73baa18ecb P d2c0e0687ac5456584fc4db0949c2f12 [term 1 FOLLOWER]: Refusing update from remote peer f7c8cbc8254d40fba61eaa0984a3fe47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:06.569358 16636 consensus_queue.cc:1035] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:06.569594 16636 consensus_queue.cc:1035] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 07:25:06.587124 16624 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 07:25:08.170615 15762 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:08.170863 15762 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:08.173638 15762 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f64d8b2626d44b29f277990aa61bf4e: Sending DeleteTablet for 3 replicas of tablet 18ddefc5dadd4c8e98011f73baa18ecb
I20260810 07:25:08.174427 15999 tablet_service.cc:1514] Processing DeleteTablet for tablet 18ddefc5dadd4c8e98011f73baa18ecb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:08 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:36656
I20260810 07:25:08.174463 16218 tablet_service.cc:1514] Processing DeleteTablet for tablet 18ddefc5dadd4c8e98011f73baa18ecb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:08 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:58630
I20260810 07:25:08.174461 16427 tablet_service.cc:1514] Processing DeleteTablet for tablet 18ddefc5dadd4c8e98011f73baa18ecb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:08 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:37718
I20260810 07:25:08.174856 16705 tablet_replica.cc:330] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47: stopping tablet replica
I20260810 07:25:08.175024 16705 raft_consensus.cc:2229] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:25:08.175392 16705 raft_consensus.cc:2258] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:08.175675 16701 tablet_replica.cc:330] T 18ddefc5dadd4c8e98011f73baa18ecb P cfc9dd8eb7bb4fcc9068037c315efa1e: stopping tablet replica
I20260810 07:25:08.175820 16701 raft_consensus.cc:2229] T 18ddefc5dadd4c8e98011f73baa18ecb P cfc9dd8eb7bb4fcc9068037c315efa1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:25:08.175948 16701 raft_consensus.cc:2258] T 18ddefc5dadd4c8e98011f73baa18ecb P cfc9dd8eb7bb4fcc9068037c315efa1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:08.176154 16705 ts_tablet_manager.cc:1905] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:08.176187 16704 tablet_replica.cc:330] T 18ddefc5dadd4c8e98011f73baa18ecb P d2c0e0687ac5456584fc4db0949c2f12: stopping tablet replica
I20260810 07:25:08.176301 16704 raft_consensus.cc:2229] T 18ddefc5dadd4c8e98011f73baa18ecb P d2c0e0687ac5456584fc4db0949c2f12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:25:08.176429 16704 raft_consensus.cc:2258] T 18ddefc5dadd4c8e98011f73baa18ecb P d2c0e0687ac5456584fc4db0949c2f12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:08.177074 16704 ts_tablet_manager.cc:1905] T 18ddefc5dadd4c8e98011f73baa18ecb P d2c0e0687ac5456584fc4db0949c2f12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:08.179040 16705 ts_tablet_manager.cc:1918] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 07:25:08.179123 16705 log.cc:1196] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-2/wal/wals/18ddefc5dadd4c8e98011f73baa18ecb
I20260810 07:25:08.179416 16705 ts_tablet_manager.cc:1939] T 18ddefc5dadd4c8e98011f73baa18ecb P f7c8cbc8254d40fba61eaa0984a3fe47: Deleting consensus metadata
I20260810 07:25:08.179968 16704 ts_tablet_manager.cc:1918] T 18ddefc5dadd4c8e98011f73baa18ecb P d2c0e0687ac5456584fc4db0949c2f12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 07:25:08.180051 16704 log.cc:1196] T 18ddefc5dadd4c8e98011f73baa18ecb P d2c0e0687ac5456584fc4db0949c2f12: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-1/wal/wals/18ddefc5dadd4c8e98011f73baa18ecb
I20260810 07:25:08.180230 15748 catalog_manager.cc:4841] TS f7c8cbc8254d40fba61eaa0984a3fe47 (127.12.157.195:34729): tablet 18ddefc5dadd4c8e98011f73baa18ecb (table test-table [id=363505947b0046e1b44c6c7bb93b050d]) successfully deleted
I20260810 07:25:08.180349 16704 ts_tablet_manager.cc:1939] T 18ddefc5dadd4c8e98011f73baa18ecb P d2c0e0687ac5456584fc4db0949c2f12: Deleting consensus metadata
I20260810 07:25:08.181154 15748 catalog_manager.cc:4841] TS d2c0e0687ac5456584fc4db0949c2f12 (127.12.157.194:46585): tablet 18ddefc5dadd4c8e98011f73baa18ecb (table test-table [id=363505947b0046e1b44c6c7bb93b050d]) successfully deleted
I20260810 07:25:08.181296 16701 ts_tablet_manager.cc:1905] T 18ddefc5dadd4c8e98011f73baa18ecb P cfc9dd8eb7bb4fcc9068037c315efa1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:08.184240 16701 ts_tablet_manager.cc:1918] T 18ddefc5dadd4c8e98011f73baa18ecb P cfc9dd8eb7bb4fcc9068037c315efa1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 07:25:08.184346 16701 log.cc:1196] T 18ddefc5dadd4c8e98011f73baa18ecb P cfc9dd8eb7bb4fcc9068037c315efa1e: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-0/wal/wals/18ddefc5dadd4c8e98011f73baa18ecb
I20260810 07:25:08.184617 16701 ts_tablet_manager.cc:1939] T 18ddefc5dadd4c8e98011f73baa18ecb P cfc9dd8eb7bb4fcc9068037c315efa1e: Deleting consensus metadata
I20260810 07:25:08.185487 15748 catalog_manager.cc:4841] TS cfc9dd8eb7bb4fcc9068037c315efa1e (127.12.157.193:41553): tablet 18ddefc5dadd4c8e98011f73baa18ecb (table test-table [id=363505947b0046e1b44c6c7bb93b050d]) successfully deleted
Aug 10 07:25:08 dist-test-slave-ffrd krb5kdc[15699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346708, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 07:25:08 dist-test-slave-ffrd krb5kdc[15699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346708, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:08.206576 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:08.197257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55158 (local address 127.12.157.254:36711)
0810 07:25:08.197577 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:08.197586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:08.197686 (+ 100us) server_negotiation.cc:407] Connection header received
0810 07:25:08.197873 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:08.197877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:08.197935 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:08.198034 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:08.198785 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:08.199359 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:08.200223 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:08.200476 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:08.200586 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:08.203133 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:08.203137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:08.203140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:08.203175 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:08.204653 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:08.205210 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:08.205215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:08.205216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:08.205308 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:08.205606 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:08.205610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:08.205612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:08.205824 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:08.205979 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:08.206240 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 07:25:08.206381 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":124,"threads_started":1}
I20260810 07:25:08.209519 15762 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:25:08.210006 15762 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:25:08.214910 16218 tablet_service.cc:1467] Processing CreateTablet for tablet 9b73f98b620e4e80b6b8728b3a4af558 (DEFAULT_TABLE table=test-table [id=d6fc8cad2b1f48a7a6659d714061d352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:08.214910 15999 tablet_service.cc:1467] Processing CreateTablet for tablet 9b73f98b620e4e80b6b8728b3a4af558 (DEFAULT_TABLE table=test-table [id=d6fc8cad2b1f48a7a6659d714061d352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:08.215009 16427 tablet_service.cc:1467] Processing CreateTablet for tablet 9b73f98b620e4e80b6b8728b3a4af558 (DEFAULT_TABLE table=test-table [id=d6fc8cad2b1f48a7a6659d714061d352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:08.215267 15999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b73f98b620e4e80b6b8728b3a4af558. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:08.215271 16218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b73f98b620e4e80b6b8728b3a4af558. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:08.215271 16427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b73f98b620e4e80b6b8728b3a4af558. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:08.218230 16718 tablet_bootstrap.cc:492] T 9b73f98b620e4e80b6b8728b3a4af558 P cfc9dd8eb7bb4fcc9068037c315efa1e: Bootstrap starting.
I20260810 07:25:08.218217 16721 tablet_bootstrap.cc:492] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12: Bootstrap starting.
I20260810 07:25:08.219444 16721 tablet_bootstrap.cc:654] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:08.219444 16718 tablet_bootstrap.cc:654] T 9b73f98b620e4e80b6b8728b3a4af558 P cfc9dd8eb7bb4fcc9068037c315efa1e: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:08.219780 16722 tablet_bootstrap.cc:492] T 9b73f98b620e4e80b6b8728b3a4af558 P f7c8cbc8254d40fba61eaa0984a3fe47: Bootstrap starting.
I20260810 07:25:08.221088 16722 tablet_bootstrap.cc:654] T 9b73f98b620e4e80b6b8728b3a4af558 P f7c8cbc8254d40fba61eaa0984a3fe47: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:08.221266 16718 tablet_bootstrap.cc:492] T 9b73f98b620e4e80b6b8728b3a4af558 P cfc9dd8eb7bb4fcc9068037c315efa1e: No bootstrap required, opened a new log
I20260810 07:25:08.221383 16718 ts_tablet_manager.cc:1397] T 9b73f98b620e4e80b6b8728b3a4af558 P cfc9dd8eb7bb4fcc9068037c315efa1e: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260810 07:25:08.221935 16718 raft_consensus.cc:348] T 9b73f98b620e4e80b6b8728b3a4af558 P cfc9dd8eb7bb4fcc9068037c315efa1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:08.222078 16718 raft_consensus.cc:374] T 9b73f98b620e4e80b6b8728b3a4af558 P cfc9dd8eb7bb4fcc9068037c315efa1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:08.222121 16718 raft_consensus.cc:729] T 9b73f98b620e4e80b6b8728b3a4af558 P cfc9dd8eb7bb4fcc9068037c315efa1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc9dd8eb7bb4fcc9068037c315efa1e, State: Initialized, Role: FOLLOWER
I20260810 07:25:08.222262 16718 consensus_queue.cc:260] T 9b73f98b620e4e80b6b8728b3a4af558 P cfc9dd8eb7bb4fcc9068037c315efa1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:08.222581 16722 tablet_bootstrap.cc:492] T 9b73f98b620e4e80b6b8728b3a4af558 P f7c8cbc8254d40fba61eaa0984a3fe47: No bootstrap required, opened a new log
I20260810 07:25:08.222678 16722 ts_tablet_manager.cc:1397] T 9b73f98b620e4e80b6b8728b3a4af558 P f7c8cbc8254d40fba61eaa0984a3fe47: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 07:25:08.222683 16718 ts_tablet_manager.cc:1428] T 9b73f98b620e4e80b6b8728b3a4af558 P cfc9dd8eb7bb4fcc9068037c315efa1e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:08.223146 16722 raft_consensus.cc:348] T 9b73f98b620e4e80b6b8728b3a4af558 P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:08.223263 16722 raft_consensus.cc:374] T 9b73f98b620e4e80b6b8728b3a4af558 P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:08.223286 16722 raft_consensus.cc:729] T 9b73f98b620e4e80b6b8728b3a4af558 P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c8cbc8254d40fba61eaa0984a3fe47, State: Initialized, Role: FOLLOWER
I20260810 07:25:08.223423 16722 consensus_queue.cc:260] T 9b73f98b620e4e80b6b8728b3a4af558 P f7c8cbc8254d40fba61eaa0984a3fe47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:08.224365 16721 tablet_bootstrap.cc:492] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12: No bootstrap required, opened a new log
I20260810 07:25:08.224488 16721 ts_tablet_manager.cc:1397] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260810 07:25:08.225054 16721 raft_consensus.cc:348] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:08.225181 16721 raft_consensus.cc:374] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:08.225237 16721 raft_consensus.cc:729] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2c0e0687ac5456584fc4db0949c2f12, State: Initialized, Role: FOLLOWER
I20260810 07:25:08.225389 16721 consensus_queue.cc:260] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:08.225852 16721 ts_tablet_manager.cc:1428] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 07:25:08.223688 16722 ts_tablet_manager.cc:1428] T 9b73f98b620e4e80b6b8728b3a4af558 P f7c8cbc8254d40fba61eaa0984a3fe47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:08.231700 16731 raft_consensus.cc:482] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:25:08.231945 16731 raft_consensus.cc:504] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:08.232398 16731 leader_election.cc:288] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfc9dd8eb7bb4fcc9068037c315efa1e (127.12.157.193:41553), f7c8cbc8254d40fba61eaa0984a3fe47 (127.12.157.195:34729)
I20260810 07:25:08.233012 16462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b73f98b620e4e80b6b8728b3a4af558" candidate_uuid: "d2c0e0687ac5456584fc4db0949c2f12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" is_pre_election: true
I20260810 07:25:08.233091 16032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b73f98b620e4e80b6b8728b3a4af558" candidate_uuid: "d2c0e0687ac5456584fc4db0949c2f12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" is_pre_election: true
I20260810 07:25:08.233180 16462 raft_consensus.cc:2454] T 9b73f98b620e4e80b6b8728b3a4af558 P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2c0e0687ac5456584fc4db0949c2f12 in term 0.
I20260810 07:25:08.233223 16032 raft_consensus.cc:2454] T 9b73f98b620e4e80b6b8728b3a4af558 P cfc9dd8eb7bb4fcc9068037c315efa1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2c0e0687ac5456584fc4db0949c2f12 in term 0.
I20260810 07:25:08.233549 16143 leader_election.cc:302] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfc9dd8eb7bb4fcc9068037c315efa1e, d2c0e0687ac5456584fc4db0949c2f12; no voters:
I20260810 07:25:08.233752 16731 raft_consensus.cc:2781] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:25:08.233816 16731 raft_consensus.cc:482] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:25:08.233865 16731 raft_consensus.cc:3037] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:08.235001 16731 raft_consensus.cc:504] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:08.235342 16731 leader_election.cc:288] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12 [CANDIDATE]: Term 1 election: Requested vote from peers cfc9dd8eb7bb4fcc9068037c315efa1e (127.12.157.193:41553), f7c8cbc8254d40fba61eaa0984a3fe47 (127.12.157.195:34729)
I20260810 07:25:08.235716 16032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b73f98b620e4e80b6b8728b3a4af558" candidate_uuid: "d2c0e0687ac5456584fc4db0949c2f12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e"
I20260810 07:25:08.235764 16462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b73f98b620e4e80b6b8728b3a4af558" candidate_uuid: "d2c0e0687ac5456584fc4db0949c2f12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47"
I20260810 07:25:08.235843 16032 raft_consensus.cc:3037] T 9b73f98b620e4e80b6b8728b3a4af558 P cfc9dd8eb7bb4fcc9068037c315efa1e [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:08.235904 16462 raft_consensus.cc:3037] T 9b73f98b620e4e80b6b8728b3a4af558 P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:08.236842 16032 raft_consensus.cc:2454] T 9b73f98b620e4e80b6b8728b3a4af558 P cfc9dd8eb7bb4fcc9068037c315efa1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2c0e0687ac5456584fc4db0949c2f12 in term 1.
I20260810 07:25:08.236920 16462 raft_consensus.cc:2454] T 9b73f98b620e4e80b6b8728b3a4af558 P f7c8cbc8254d40fba61eaa0984a3fe47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2c0e0687ac5456584fc4db0949c2f12 in term 1.
I20260810 07:25:08.237188 16143 leader_election.cc:302] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfc9dd8eb7bb4fcc9068037c315efa1e, d2c0e0687ac5456584fc4db0949c2f12; no voters:
I20260810 07:25:08.237378 16731 raft_consensus.cc:2781] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:08.237589 16731 raft_consensus.cc:686] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12 [term 1 LEADER]: Becoming Leader. State: Replica: d2c0e0687ac5456584fc4db0949c2f12, State: Running, Role: LEADER
I20260810 07:25:08.237744 16731 consensus_queue.cc:237] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:08.239521 15762 catalog_manager.cc:5495] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12 reported cstate change: term changed from 0 to 1, leader changed from <none> to d2c0e0687ac5456584fc4db0949c2f12 (127.12.157.194). New cstate: current_term: 1 leader_uuid: "d2c0e0687ac5456584fc4db0949c2f12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } health_report { overall_health: UNKNOWN } } }
I20260810 07:25:08.259259 16736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:08.255204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46632 (local address 127.12.157.194:46585)
0810 07:25:08.255527 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:08.255548 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:08.255560 (+ 12us) server_negotiation.cc:407] Connection header received
0810 07:25:08.255605 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:08.255607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:08.255647 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:08.255738 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:25:08.256026 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:08.256669 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:08.257676 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:08.257927 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:08.258037 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:08.258360 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:25:08.258446 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:25:08.258526 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:08.258633 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 07:25:08.258687 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":110,"threads_started":1}
I20260810 07:25:08.266579 16462 raft_consensus.cc:1261] T 9b73f98b620e4e80b6b8728b3a4af558 P f7c8cbc8254d40fba61eaa0984a3fe47 [term 1 FOLLOWER]: Refusing update from remote peer d2c0e0687ac5456584fc4db0949c2f12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:08.266669 16032 raft_consensus.cc:1261] T 9b73f98b620e4e80b6b8728b3a4af558 P cfc9dd8eb7bb4fcc9068037c315efa1e [term 1 FOLLOWER]: Refusing update from remote peer d2c0e0687ac5456584fc4db0949c2f12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:08.267068 16732 consensus_queue.cc:1035] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:08.267263 16731 consensus_queue.cc:1035] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:08.301321 15766 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:08.301479 15766 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:08.303241 15766 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f64d8b2626d44b29f277990aa61bf4e: Sending DeleteTablet for 3 replicas of tablet 9b73f98b620e4e80b6b8728b3a4af558
I20260810 07:25:08.303767 15999 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b73f98b620e4e80b6b8728b3a4af558 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:08 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:36656
I20260810 07:25:08.303812 16218 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b73f98b620e4e80b6b8728b3a4af558 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:08 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:58630
I20260810 07:25:08.303920 16427 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b73f98b620e4e80b6b8728b3a4af558 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:08 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:37718
I20260810 07:25:08.303958 16701 tablet_replica.cc:330] T 9b73f98b620e4e80b6b8728b3a4af558 P cfc9dd8eb7bb4fcc9068037c315efa1e: stopping tablet replica
I20260810 07:25:08.304021 16704 tablet_replica.cc:330] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12: stopping tablet replica
I20260810 07:25:08.304082 16701 raft_consensus.cc:2229] T 9b73f98b620e4e80b6b8728b3a4af558 P cfc9dd8eb7bb4fcc9068037c315efa1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:25:08.304155 16704 raft_consensus.cc:2229] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:25:08.304246 16701 raft_consensus.cc:2258] T 9b73f98b620e4e80b6b8728b3a4af558 P cfc9dd8eb7bb4fcc9068037c315efa1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:08.304330 16704 raft_consensus.cc:2258] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:08.304419 16705 tablet_replica.cc:330] T 9b73f98b620e4e80b6b8728b3a4af558 P f7c8cbc8254d40fba61eaa0984a3fe47: stopping tablet replica
I20260810 07:25:08.304553 16705 raft_consensus.cc:2229] T 9b73f98b620e4e80b6b8728b3a4af558 P f7c8cbc8254d40fba61eaa0984a3fe47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:25:08.304665 16705 raft_consensus.cc:2258] T 9b73f98b620e4e80b6b8728b3a4af558 P f7c8cbc8254d40fba61eaa0984a3fe47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:08.305070 16704 ts_tablet_manager.cc:1905] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:08.305326 15766 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55158:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 07:25:08.305467 16701 ts_tablet_manager.cc:1905] T 9b73f98b620e4e80b6b8728b3a4af558 P cfc9dd8eb7bb4fcc9068037c315efa1e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 07:25:08.305651 15766 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:25:08.306933 16705 ts_tablet_manager.cc:1905] T 9b73f98b620e4e80b6b8728b3a4af558 P f7c8cbc8254d40fba61eaa0984a3fe47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:08.308138 16704 ts_tablet_manager.cc:1918] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 07:25:08.308138 16701 ts_tablet_manager.cc:1918] T 9b73f98b620e4e80b6b8728b3a4af558 P cfc9dd8eb7bb4fcc9068037c315efa1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 07:25:08.308211 16704 log.cc:1196] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-1/wal/wals/9b73f98b620e4e80b6b8728b3a4af558
I20260810 07:25:08.308211 16701 log.cc:1196] T 9b73f98b620e4e80b6b8728b3a4af558 P cfc9dd8eb7bb4fcc9068037c315efa1e: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-0/wal/wals/9b73f98b620e4e80b6b8728b3a4af558
I20260810 07:25:08.308493 16704 ts_tablet_manager.cc:1939] T 9b73f98b620e4e80b6b8728b3a4af558 P d2c0e0687ac5456584fc4db0949c2f12: Deleting consensus metadata
I20260810 07:25:08.308499 16701 ts_tablet_manager.cc:1939] T 9b73f98b620e4e80b6b8728b3a4af558 P cfc9dd8eb7bb4fcc9068037c315efa1e: Deleting consensus metadata
I20260810 07:25:08.309216 15748 catalog_manager.cc:4841] TS cfc9dd8eb7bb4fcc9068037c315efa1e (127.12.157.193:41553): tablet 9b73f98b620e4e80b6b8728b3a4af558 (table test-table [id=d6fc8cad2b1f48a7a6659d714061d352]) successfully deleted
I20260810 07:25:08.309471 15748 catalog_manager.cc:4841] TS d2c0e0687ac5456584fc4db0949c2f12 (127.12.157.194:46585): tablet 9b73f98b620e4e80b6b8728b3a4af558 (table test-table [id=d6fc8cad2b1f48a7a6659d714061d352]) successfully deleted
I20260810 07:25:08.309772 16705 ts_tablet_manager.cc:1918] T 9b73f98b620e4e80b6b8728b3a4af558 P f7c8cbc8254d40fba61eaa0984a3fe47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 07:25:08.309863 16705 log.cc:1196] T 9b73f98b620e4e80b6b8728b3a4af558 P f7c8cbc8254d40fba61eaa0984a3fe47: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-2/wal/wals/9b73f98b620e4e80b6b8728b3a4af558
I20260810 07:25:08.310109 16705 ts_tablet_manager.cc:1939] T 9b73f98b620e4e80b6b8728b3a4af558 P f7c8cbc8254d40fba61eaa0984a3fe47: Deleting consensus metadata
I20260810 07:25:08.310706 15748 catalog_manager.cc:4841] TS f7c8cbc8254d40fba61eaa0984a3fe47 (127.12.157.195:34729): tablet 9b73f98b620e4e80b6b8728b3a4af558 (table test-table [id=d6fc8cad2b1f48a7a6659d714061d352]) successfully deleted
I20260810 07:25:08.313942 16218 tablet_service.cc:1467] Processing CreateTablet for tablet 1159541ee4604c4c9b8b2e6dcc4639ca (DEFAULT_TABLE table=test-table [id=2f54cd2f0f464c95b8f07661ba963755]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:08.313920 15999 tablet_service.cc:1467] Processing CreateTablet for tablet 1159541ee4604c4c9b8b2e6dcc4639ca (DEFAULT_TABLE table=test-table [id=2f54cd2f0f464c95b8f07661ba963755]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:08.314190 16218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1159541ee4604c4c9b8b2e6dcc4639ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:08.314189 15999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1159541ee4604c4c9b8b2e6dcc4639ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:08.316629 16721 tablet_bootstrap.cc:492] T 1159541ee4604c4c9b8b2e6dcc4639ca P d2c0e0687ac5456584fc4db0949c2f12: Bootstrap starting.
I20260810 07:25:08.316720 16718 tablet_bootstrap.cc:492] T 1159541ee4604c4c9b8b2e6dcc4639ca P cfc9dd8eb7bb4fcc9068037c315efa1e: Bootstrap starting.
I20260810 07:25:08.317317 16427 tablet_service.cc:1467] Processing CreateTablet for tablet 1159541ee4604c4c9b8b2e6dcc4639ca (DEFAULT_TABLE table=test-table [id=2f54cd2f0f464c95b8f07661ba963755]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:08.317601 16721 tablet_bootstrap.cc:654] T 1159541ee4604c4c9b8b2e6dcc4639ca P d2c0e0687ac5456584fc4db0949c2f12: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:08.317620 16427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1159541ee4604c4c9b8b2e6dcc4639ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:08.317824 16718 tablet_bootstrap.cc:654] T 1159541ee4604c4c9b8b2e6dcc4639ca P cfc9dd8eb7bb4fcc9068037c315efa1e: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:08.318938 16721 tablet_bootstrap.cc:492] T 1159541ee4604c4c9b8b2e6dcc4639ca P d2c0e0687ac5456584fc4db0949c2f12: No bootstrap required, opened a new log
I20260810 07:25:08.319010 16721 ts_tablet_manager.cc:1397] T 1159541ee4604c4c9b8b2e6dcc4639ca P d2c0e0687ac5456584fc4db0949c2f12: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 07:25:08.319352 16721 raft_consensus.cc:348] T 1159541ee4604c4c9b8b2e6dcc4639ca P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:08.319487 16721 raft_consensus.cc:374] T 1159541ee4604c4c9b8b2e6dcc4639ca P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:08.319523 16721 raft_consensus.cc:729] T 1159541ee4604c4c9b8b2e6dcc4639ca P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2c0e0687ac5456584fc4db0949c2f12, State: Initialized, Role: FOLLOWER
I20260810 07:25:08.319676 16721 consensus_queue.cc:260] T 1159541ee4604c4c9b8b2e6dcc4639ca P d2c0e0687ac5456584fc4db0949c2f12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:08.319857 16718 tablet_bootstrap.cc:492] T 1159541ee4604c4c9b8b2e6dcc4639ca P cfc9dd8eb7bb4fcc9068037c315efa1e: No bootstrap required, opened a new log
I20260810 07:25:08.319953 16718 ts_tablet_manager.cc:1397] T 1159541ee4604c4c9b8b2e6dcc4639ca P cfc9dd8eb7bb4fcc9068037c315efa1e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 07:25:08.319998 16721 ts_tablet_manager.cc:1428] T 1159541ee4604c4c9b8b2e6dcc4639ca P d2c0e0687ac5456584fc4db0949c2f12: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:08.320561 16718 raft_consensus.cc:348] T 1159541ee4604c4c9b8b2e6dcc4639ca P cfc9dd8eb7bb4fcc9068037c315efa1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:08.320657 16722 tablet_bootstrap.cc:492] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47: Bootstrap starting.
I20260810 07:25:08.320679 16718 raft_consensus.cc:374] T 1159541ee4604c4c9b8b2e6dcc4639ca P cfc9dd8eb7bb4fcc9068037c315efa1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:08.320719 16718 raft_consensus.cc:729] T 1159541ee4604c4c9b8b2e6dcc4639ca P cfc9dd8eb7bb4fcc9068037c315efa1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfc9dd8eb7bb4fcc9068037c315efa1e, State: Initialized, Role: FOLLOWER
I20260810 07:25:08.320874 16718 consensus_queue.cc:260] T 1159541ee4604c4c9b8b2e6dcc4639ca P cfc9dd8eb7bb4fcc9068037c315efa1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:08.321188 16718 ts_tablet_manager.cc:1428] T 1159541ee4604c4c9b8b2e6dcc4639ca P cfc9dd8eb7bb4fcc9068037c315efa1e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 07:25:08.321792 16722 tablet_bootstrap.cc:654] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:08.323047 16722 tablet_bootstrap.cc:492] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47: No bootstrap required, opened a new log
I20260810 07:25:08.323170 16722 ts_tablet_manager.cc:1397] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 07:25:08.323647 16722 raft_consensus.cc:348] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:08.323801 16722 raft_consensus.cc:374] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:08.323850 16722 raft_consensus.cc:729] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7c8cbc8254d40fba61eaa0984a3fe47, State: Initialized, Role: FOLLOWER
I20260810 07:25:08.324015 16722 consensus_queue.cc:260] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:08.324349 16722 ts_tablet_manager.cc:1428] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 07:25:08.687474 16673 raft_consensus.cc:482] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:25:08.687638 16673 raft_consensus.cc:504] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:08.688016 16673 leader_election.cc:288] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cfc9dd8eb7bb4fcc9068037c315efa1e (127.12.157.193:41553), d2c0e0687ac5456584fc4db0949c2f12 (127.12.157.194:46585)
I20260810 07:25:08.688645 16247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1159541ee4604c4c9b8b2e6dcc4639ca" candidate_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2c0e0687ac5456584fc4db0949c2f12" is_pre_election: true
I20260810 07:25:08.688649 16034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1159541ee4604c4c9b8b2e6dcc4639ca" candidate_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" is_pre_election: true
I20260810 07:25:08.688791 16247 raft_consensus.cc:2454] T 1159541ee4604c4c9b8b2e6dcc4639ca P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c8cbc8254d40fba61eaa0984a3fe47 in term 0.
I20260810 07:25:08.688823 16034 raft_consensus.cc:2454] T 1159541ee4604c4c9b8b2e6dcc4639ca P cfc9dd8eb7bb4fcc9068037c315efa1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7c8cbc8254d40fba61eaa0984a3fe47 in term 0.
I20260810 07:25:08.689236 16358 leader_election.cc:302] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cfc9dd8eb7bb4fcc9068037c315efa1e, f7c8cbc8254d40fba61eaa0984a3fe47; no voters:
I20260810 07:25:08.689513 16673 raft_consensus.cc:2781] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:25:08.689582 16673 raft_consensus.cc:482] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:25:08.689612 16673 raft_consensus.cc:3037] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:08.690708 16673 raft_consensus.cc:504] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:08.691056 16673 leader_election.cc:288] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47 [CANDIDATE]: Term 1 election: Requested vote from peers cfc9dd8eb7bb4fcc9068037c315efa1e (127.12.157.193:41553), d2c0e0687ac5456584fc4db0949c2f12 (127.12.157.194:46585)
I20260810 07:25:08.691567 16247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1159541ee4604c4c9b8b2e6dcc4639ca" candidate_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2c0e0687ac5456584fc4db0949c2f12"
I20260810 07:25:08.691620 16034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1159541ee4604c4c9b8b2e6dcc4639ca" candidate_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e"
I20260810 07:25:08.691679 16247 raft_consensus.cc:3037] T 1159541ee4604c4c9b8b2e6dcc4639ca P d2c0e0687ac5456584fc4db0949c2f12 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:08.691800 16034 raft_consensus.cc:3037] T 1159541ee4604c4c9b8b2e6dcc4639ca P cfc9dd8eb7bb4fcc9068037c315efa1e [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:08.692776 16247 raft_consensus.cc:2454] T 1159541ee4604c4c9b8b2e6dcc4639ca P d2c0e0687ac5456584fc4db0949c2f12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c8cbc8254d40fba61eaa0984a3fe47 in term 1.
I20260810 07:25:08.692978 16034 raft_consensus.cc:2454] T 1159541ee4604c4c9b8b2e6dcc4639ca P cfc9dd8eb7bb4fcc9068037c315efa1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7c8cbc8254d40fba61eaa0984a3fe47 in term 1.
I20260810 07:25:08.693137 16358 leader_election.cc:302] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2c0e0687ac5456584fc4db0949c2f12, f7c8cbc8254d40fba61eaa0984a3fe47; no voters:
I20260810 07:25:08.693301 16673 raft_consensus.cc:2781] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:08.693408 16673 raft_consensus.cc:686] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47 [term 1 LEADER]: Becoming Leader. State: Replica: f7c8cbc8254d40fba61eaa0984a3fe47, State: Running, Role: LEADER
I20260810 07:25:08.693567 16673 consensus_queue.cc:237] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } }
I20260810 07:25:08.695384 15764 catalog_manager.cc:5495] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7c8cbc8254d40fba61eaa0984a3fe47 (127.12.157.195). New cstate: current_term: 1 leader_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7c8cbc8254d40fba61eaa0984a3fe47" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34729 } health_report { overall_health: HEALTHY } } }
I20260810 07:25:08.781440 16763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:08.778185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59212 (local address 127.12.157.195:34729)
0810 07:25:08.778467 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:08.778474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:08.778486 (+ 12us) server_negotiation.cc:407] Connection header received
0810 07:25:08.778524 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:08.778526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:08.778567 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:08.778637 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:25:08.778879 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:08.779395 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:08.780323 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:08.780561 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:08.780654 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:08.780894 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:25:08.781038 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:25:08.781102 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:08.781192 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 07:25:08.781246 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":118,"threads_started":1}
I20260810 07:25:08.787386 16034 raft_consensus.cc:1261] T 1159541ee4604c4c9b8b2e6dcc4639ca P cfc9dd8eb7bb4fcc9068037c315efa1e [term 1 FOLLOWER]: Refusing update from remote peer f7c8cbc8254d40fba61eaa0984a3fe47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:08.787400 16247 raft_consensus.cc:1261] T 1159541ee4604c4c9b8b2e6dcc4639ca P d2c0e0687ac5456584fc4db0949c2f12 [term 1 FOLLOWER]: Refusing update from remote peer f7c8cbc8254d40fba61eaa0984a3fe47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:08.787792 16673 consensus_queue.cc:1035] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2c0e0687ac5456584fc4db0949c2f12" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:08.787967 16649 consensus_queue.cc:1035] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfc9dd8eb7bb4fcc9068037c315efa1e" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 07:25:08.805536 16670 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 07:25:10.357043 15766 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:10.357225 15766 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55158:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:10.359862 15766 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f64d8b2626d44b29f277990aa61bf4e: Sending DeleteTablet for 3 replicas of tablet 1159541ee4604c4c9b8b2e6dcc4639ca
I20260810 07:25:10.360633 16427 tablet_service.cc:1514] Processing DeleteTablet for tablet 1159541ee4604c4c9b8b2e6dcc4639ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:10 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:37718
I20260810 07:25:10.360635 16218 tablet_service.cc:1514] Processing DeleteTablet for tablet 1159541ee4604c4c9b8b2e6dcc4639ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:10 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:58630
I20260810 07:25:10.360872 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 15891
I20260810 07:25:10.369663 16806 tablet_replica.cc:330] T 1159541ee4604c4c9b8b2e6dcc4639ca P d2c0e0687ac5456584fc4db0949c2f12: stopping tablet replica
I20260810 07:25:10.369863 16806 raft_consensus.cc:2229] T 1159541ee4604c4c9b8b2e6dcc4639ca P d2c0e0687ac5456584fc4db0949c2f12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:25:10.370026 16806 raft_consensus.cc:2258] T 1159541ee4604c4c9b8b2e6dcc4639ca P d2c0e0687ac5456584fc4db0949c2f12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:10.370401 16803 tablet_replica.cc:330] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47: stopping tablet replica
I20260810 07:25:10.370611 16803 raft_consensus.cc:2229] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:25:10.370849 16806 ts_tablet_manager.cc:1905] T 1159541ee4604c4c9b8b2e6dcc4639ca P d2c0e0687ac5456584fc4db0949c2f12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:10.370898 16803 raft_consensus.cc:2258] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:10.371719 16803 ts_tablet_manager.cc:1905] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:10.374236 16806 ts_tablet_manager.cc:1918] T 1159541ee4604c4c9b8b2e6dcc4639ca P d2c0e0687ac5456584fc4db0949c2f12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 07:25:10.374339 16806 log.cc:1196] T 1159541ee4604c4c9b8b2e6dcc4639ca P d2c0e0687ac5456584fc4db0949c2f12: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-1/wal/wals/1159541ee4604c4c9b8b2e6dcc4639ca
I20260810 07:25:10.374650 16806 ts_tablet_manager.cc:1939] T 1159541ee4604c4c9b8b2e6dcc4639ca P d2c0e0687ac5456584fc4db0949c2f12: Deleting consensus metadata
I20260810 07:25:10.374912 16803 ts_tablet_manager.cc:1918] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 07:25:10.375008 16803 log.cc:1196] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786346694975122-12919-0/minicluster-data/ts-2/wal/wals/1159541ee4604c4c9b8b2e6dcc4639ca
I20260810 07:25:10.375314 16803 ts_tablet_manager.cc:1939] T 1159541ee4604c4c9b8b2e6dcc4639ca P f7c8cbc8254d40fba61eaa0984a3fe47: Deleting consensus metadata
I20260810 07:25:10.375599 15748 catalog_manager.cc:4841] TS d2c0e0687ac5456584fc4db0949c2f12 (127.12.157.194:46585): tablet 1159541ee4604c4c9b8b2e6dcc4639ca (table test-table [id=2f54cd2f0f464c95b8f07661ba963755]) successfully deleted
I20260810 07:25:10.376029 15748 catalog_manager.cc:4841] TS f7c8cbc8254d40fba61eaa0984a3fe47 (127.12.157.195:34729): tablet 1159541ee4604c4c9b8b2e6dcc4639ca (table test-table [id=2f54cd2f0f464c95b8f07661ba963755]) successfully deleted
W20260810 07:25:10.378263 15748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.157.193:41553 (error 108)
I20260810 07:25:10.379078 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 16113
I20260810 07:25:10.388628 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:10.379171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:41553 (local address 127.0.0.1:59168)
0810 07:25:10.388428 (+ 9257us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:10.388529 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.193:41553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9159,"thread_start_us":89,"threads_started":1}
W20260810 07:25:10.388882 15748 catalog_manager.cc:4584] TS cfc9dd8eb7bb4fcc9068037c315efa1e (127.12.157.193:41553): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1159541ee4604c4c9b8b2e6dcc4639ca: Network error: Client connection negotiation failed: client connection to 127.12.157.193:41553: connect: Connection refused (error 111)
I20260810 07:25:10.395880 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 16334
I20260810 07:25:10.412791 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 15724
2026-08-10T07:25:10Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6801 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:25:10 dist-test-slave-ffrd krb5kdc[16818](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:25:10 dist-test-slave-ffrd krb5kdc[16818](info): set up 2 sockets
Aug 10 07:25:10 dist-test-slave-ffrd krb5kdc[16818](info): commencing operation
krb5kdc: starting...
W20260810 07:25:11.719233 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.276s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:25:11 dist-test-slave-ffrd krb5kdc[16818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:25:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:25:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:25:11.857257 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:37857
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:34301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:37857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:11.976616 16838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:11.976920 16838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:11.977049 16838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:11.981179 16838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:25:11.981256 16838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:11.981276 16838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:11.981292 16838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:25:11.981308 16838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:25:11.986845 16838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:34301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:37857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:37857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:11.988019 16838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:11.989172 16838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:11.999143 16851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:11.999197 16838 server_base.cc:1034] running on GCE node
W20260810 07:25:11.999146 16848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:11.999172 16845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:12.000120 16838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:12.001320 16838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:12.002640 16838 hybrid_clock.cc:648] HybridClock initialized: now 1786346712002622 us; error 43 us; skew 500 ppm
Aug 10 07:25:12 dist-test-slave-ffrd krb5kdc[16818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346712, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:12.006057 16838 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:25:12.007051 16838 webserver.cc:460] Webserver started at http://127.12.157.254:37385/ using document root <none> and password file <none>
I20260810 07:25:12.007635 16838 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:12.007701 16838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:12.007890 16838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:12.010267 16838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "d9ef576e58a94c20b78fc5172470f215"
format_stamp: "Formatted at 2026-08-10 07:25:12 on dist-test-slave-ffrd"
server_key: "24c4cd3c17c2c6f2319cfbe1978de2f6"
server_key_iv: "aff1b621172ea9d264699ceeb8423ce1"
server_key_version: "encryptionkey@0"
I20260810 07:25:12.010877 16838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "d9ef576e58a94c20b78fc5172470f215"
format_stamp: "Formatted at 2026-08-10 07:25:12 on dist-test-slave-ffrd"
server_key: "24c4cd3c17c2c6f2319cfbe1978de2f6"
server_key_iv: "aff1b621172ea9d264699ceeb8423ce1"
server_key_version: "encryptionkey@0"
I20260810 07:25:12.015177 16838 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 07:25:12.017905 16861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:12.019357 16838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:25:12.019474 16838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "d9ef576e58a94c20b78fc5172470f215"
format_stamp: "Formatted at 2026-08-10 07:25:12 on dist-test-slave-ffrd"
server_key: "24c4cd3c17c2c6f2319cfbe1978de2f6"
server_key_iv: "aff1b621172ea9d264699ceeb8423ce1"
server_key_version: "encryptionkey@0"
I20260810 07:25:12.019572 16838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:12.039377 16838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:12.043183 16838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:12.043426 16838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:12.052001 16838 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:37857
I20260810 07:25:12.052029 16943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:37857 every 8 connection(s)
I20260810 07:25:12.053076 16838 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:25:12.055461 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 16838
I20260810 07:25:12.055554 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:25:12.055869 12919 external_mini_cluster.cc:1428] Setting key 0eeee7163de8ecd81bb6d1cbbda7c8dc
I20260810 07:25:12.056048 16944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:12.062613 16944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9ef576e58a94c20b78fc5172470f215: Bootstrap starting.
I20260810 07:25:12.065092 16944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9ef576e58a94c20b78fc5172470f215: Neither blocks nor log segments found. Creating new log.
Aug 10 07:25:12 dist-test-slave-ffrd krb5kdc[16818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:12.065927 16944 log.cc:824] T 00000000000000000000000000000000 P d9ef576e58a94c20b78fc5172470f215: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:12.068380 16944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9ef576e58a94c20b78fc5172470f215: No bootstrap required, opened a new log
I20260810 07:25:12.072345 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:12.057426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35946 (local address 127.12.157.254:37857)
0810 07:25:12.057907 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:12.057924 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:12.057963 (+ 39us) server_negotiation.cc:407] Connection header received
0810 07:25:12.058909 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:12.058934 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:12.059281 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:12.059667 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:12.060596 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:12.061789 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:12.062771 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:12.063291 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:12.063487 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:12.066462 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:12.066468 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:12.066482 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:12.066513 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:12.068689 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:12.069281 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:12.069287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:12.069294 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:12.069450 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:12.069751 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:12.069755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:12.069757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:12.070309 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:12.070530 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:12.070760 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 07:25:12.071155 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":132,"threads_started":1}
I20260810 07:25:12.073604 16944 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9ef576e58a94c20b78fc5172470f215 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ef576e58a94c20b78fc5172470f215" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37857 } }
I20260810 07:25:12.073886 16944 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9ef576e58a94c20b78fc5172470f215 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:12.073949 16944 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9ef576e58a94c20b78fc5172470f215 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9ef576e58a94c20b78fc5172470f215, State: Initialized, Role: FOLLOWER
I20260810 07:25:12.074421 16944 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9ef576e58a94c20b78fc5172470f215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ef576e58a94c20b78fc5172470f215" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37857 } }
I20260810 07:25:12.074579 16944 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9ef576e58a94c20b78fc5172470f215 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:25:12.074648 16944 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9ef576e58a94c20b78fc5172470f215 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:25:12.074746 16944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9ef576e58a94c20b78fc5172470f215 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:12.075728 16944 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9ef576e58a94c20b78fc5172470f215 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ef576e58a94c20b78fc5172470f215" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37857 } }
I20260810 07:25:12.076076 16944 leader_election.cc:302] T 00000000000000000000000000000000 P d9ef576e58a94c20b78fc5172470f215 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9ef576e58a94c20b78fc5172470f215; no voters:
I20260810 07:25:12.076403 16944 leader_election.cc:288] T 00000000000000000000000000000000 P d9ef576e58a94c20b78fc5172470f215 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:25:12.076627 16954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9ef576e58a94c20b78fc5172470f215 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:12.076872 16954 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9ef576e58a94c20b78fc5172470f215 [term 1 LEADER]: Becoming Leader. State: Replica: d9ef576e58a94c20b78fc5172470f215, State: Running, Role: LEADER
I20260810 07:25:12.077251 16954 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9ef576e58a94c20b78fc5172470f215 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ef576e58a94c20b78fc5172470f215" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37857 } }
I20260810 07:25:12.077706 16944 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9ef576e58a94c20b78fc5172470f215 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:25:12.078974 16956 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9ef576e58a94c20b78fc5172470f215 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9ef576e58a94c20b78fc5172470f215" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ef576e58a94c20b78fc5172470f215" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37857 } } }
I20260810 07:25:12.079104 16956 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9ef576e58a94c20b78fc5172470f215 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:12.079403 16957 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9ef576e58a94c20b78fc5172470f215 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9ef576e58a94c20b78fc5172470f215. Latest consensus state: current_term: 1 leader_uuid: "d9ef576e58a94c20b78fc5172470f215" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9ef576e58a94c20b78fc5172470f215" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37857 } } }
I20260810 07:25:12.079494 16957 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9ef576e58a94c20b78fc5172470f215 [sys.catalog]: This master's current role is: LEADER
W20260810 07:25:12.082718 16973 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d9ef576e58a94c20b78fc5172470f215: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:25:12.082819 16973 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:25:12.082898 16965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:25:12.083599 16965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:25:12.088901 16965 catalog_manager.cc:1347] Generated new cluster ID: ec2bd2fab74e487686aa785c50118132
I20260810 07:25:12.088999 16965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:25:12.107839 16965 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:25:12.109242 16965 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:25:12.116847 16965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9ef576e58a94c20b78fc5172470f215: Generated new TSK 0
I20260810 07:25:12.117637 16965 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
I20260810 07:25:12.156682 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:0
--local_ip_for_outbound_sockets=127.12.157.193
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37857
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:34301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:12.277263 16990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:12.277570 16990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:12.277652 16990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:12.281992 16990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:12.282085 16990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:12.282179 16990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:25:12.287518 16990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:34301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:12.288667 16990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:12.289788 16990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 07:25:12.300405 16990 server_base.cc:1034] running on GCE node
W20260810 07:25:12.300338 17004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:12.300351 17002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:12.300392 17000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:12.301080 16990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:12.301714 16990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:12.302902 16990 hybrid_clock.cc:648] HybridClock initialized: now 1786346712302865 us; error 40 us; skew 500 ppm
Aug 10 07:25:12 dist-test-slave-ffrd krb5kdc[16818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346712, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:12.306075 16990 init.cc:377] Logged in from keytab as kudu/127.12.157.193@KRBTEST.COM (short username kudu)
I20260810 07:25:12.306963 16990 webserver.cc:460] Webserver started at http://127.12.157.193:42725/ using document root <none> and password file <none>
I20260810 07:25:12.307539 16990 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:12.307600 16990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:12.307843 16990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:12.310037 16990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/data/instance:
uuid: "f72cff4f66014af19c2e24787a0482c4"
format_stamp: "Formatted at 2026-08-10 07:25:12 on dist-test-slave-ffrd"
server_key: "94a8a5730998bf7e69c96c1786026c08"
server_key_iv: "8a486d6ab64fafcec907697ecf8fc8a2"
server_key_version: "encryptionkey@0"
I20260810 07:25:12.310619 16990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance:
uuid: "f72cff4f66014af19c2e24787a0482c4"
format_stamp: "Formatted at 2026-08-10 07:25:12 on dist-test-slave-ffrd"
server_key: "94a8a5730998bf7e69c96c1786026c08"
server_key_iv: "8a486d6ab64fafcec907697ecf8fc8a2"
server_key_version: "encryptionkey@0"
I20260810 07:25:12.314594 16990 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 07:25:12.317245 17017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:12.318452 16990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:25:12.318583 16990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "f72cff4f66014af19c2e24787a0482c4"
format_stamp: "Formatted at 2026-08-10 07:25:12 on dist-test-slave-ffrd"
server_key: "94a8a5730998bf7e69c96c1786026c08"
server_key_iv: "8a486d6ab64fafcec907697ecf8fc8a2"
server_key_version: "encryptionkey@0"
I20260810 07:25:12.318697 16990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:12.331751 16990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:12.335224 16990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:12.335451 16990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:12.336081 16990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:12.336971 16990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:12.337020 16990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:12.337091 16990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:12.337123 16990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:12.346700 16990 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:44969
I20260810 07:25:12.346727 17206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:44969 every 8 connection(s)
I20260810 07:25:12.347607 16990 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:25:12.354223 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 16990
I20260810 07:25:12.354318 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance
I20260810 07:25:12.354600 12919 external_mini_cluster.cc:1428] Setting key be828f5923b2955443e3463dac284622
Aug 10 07:25:12 dist-test-slave-ffrd krb5kdc[16818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346712, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:12.361680 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:12.349543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:56297 (local address 127.12.157.254:37857)
0810 07:25:12.349687 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:12.349690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:12.350668 (+ 978us) server_negotiation.cc:407] Connection header received
0810 07:25:12.351567 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:12.351571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:12.351631 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:12.351744 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:12.352974 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:12.353523 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:12.354811 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:12.355050 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:12.355169 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:12.357795 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:12.357798 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:12.357800 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:12.357823 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:12.359258 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:12.359879 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:12.359883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:12.359885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:12.359968 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:12.360366 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:12.360368 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:12.360369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:12.360565 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:12.360645 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:12.361452 (+ 807us) server_negotiation.cc:299] Negotiation successful
0810 07:25:12.361560 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 07:25:12.362835 17212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:12.349855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37857 (local address 127.12.157.193:56297)
0810 07:25:12.350516 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:12.350558 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:12.351362 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:12.351918 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:12.351925 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:12.352273 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:12.352795 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:12.352809 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:12.353657 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:12.353661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:12.354678 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:12.354693 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:12.355214 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:12.355218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:12.355396 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:12.356090 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:12.356107 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:12.357620 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:12.359385 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:12.359393 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:12.359411 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:12.359770 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:12.360071 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:12.360077 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:12.360082 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:12.360269 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:12.360674 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:12.360683 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:12.361095 (+ 412us) client_negotiation.cc:769] Sending connection context
0810 07:25:12.361328 (+ 233us) client_negotiation.cc:240] Negotiation successful
0810 07:25:12.361707 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":112,"threads_started":1}
I20260810 07:25:12.364099 17207 heartbeater.cc:347] Connected to a master server at 127.12.157.254:37857
I20260810 07:25:12.364367 17207 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:12.364982 17207 heartbeater.cc:511] Master 127.12.157.254:37857 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.157.194@KRBTEST.COM; defaulting to no policy
I20260810 07:25:12.366565 16888 ts_manager.cc:194] Registered new tserver with Master: f72cff4f66014af19c2e24787a0482c4 (127.12.157.193:44969)
I20260810 07:25:12.367787 16888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.193@KRBTEST.COM'} at 127.12.157.193:56297
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
I20260810 07:25:12.383919 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:0
--local_ip_for_outbound_sockets=127.12.157.194
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37857
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:34301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:12.501159 17217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:12.501400 17217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:12.501443 17217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:12.505306 17217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:12.505352 17217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:12.505407 17217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:25:12.510375 17217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:34301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:12.511300 17217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:12.512244 17217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:12.521726 17231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:12.521740 17230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:12.521766 17233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:12.522272 17217 server_base.cc:1034] running on GCE node
I20260810 07:25:12.522692 17217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:12.523468 17217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:12.524641 17217 hybrid_clock.cc:648] HybridClock initialized: now 1786346712524616 us; error 32 us; skew 500 ppm
Aug 10 07:25:12 dist-test-slave-ffrd krb5kdc[16818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346712, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:12.527470 17217 init.cc:377] Logged in from keytab as kudu/127.12.157.194@KRBTEST.COM (short username kudu)
I20260810 07:25:12.528307 17217 webserver.cc:460] Webserver started at http://127.12.157.194:33963/ using document root <none> and password file <none>
I20260810 07:25:12.528869 17217 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:12.528999 17217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:12.529255 17217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:12.531279 17217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/data/instance:
uuid: "f3b623630c71429ca3610f7a10ee6815"
format_stamp: "Formatted at 2026-08-10 07:25:12 on dist-test-slave-ffrd"
server_key: "0d012681f8d36ab642d82cb9aa6857d6"
server_key_iv: "dc7b0d0a47ac2037d82e9447c31214f4"
server_key_version: "encryptionkey@0"
I20260810 07:25:12.531857 17217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance:
uuid: "f3b623630c71429ca3610f7a10ee6815"
format_stamp: "Formatted at 2026-08-10 07:25:12 on dist-test-slave-ffrd"
server_key: "0d012681f8d36ab642d82cb9aa6857d6"
server_key_iv: "dc7b0d0a47ac2037d82e9447c31214f4"
server_key_version: "encryptionkey@0"
I20260810 07:25:12.535646 17217 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 07:25:12.538139 17245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:12.539395 17217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 07:25:12.539537 17217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "f3b623630c71429ca3610f7a10ee6815"
format_stamp: "Formatted at 2026-08-10 07:25:12 on dist-test-slave-ffrd"
server_key: "0d012681f8d36ab642d82cb9aa6857d6"
server_key_iv: "dc7b0d0a47ac2037d82e9447c31214f4"
server_key_version: "encryptionkey@0"
I20260810 07:25:12.539649 17217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:12.553319 17217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:12.556246 17217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:12.556463 17217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:12.557157 17217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:12.558000 17217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:12.558048 17217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:12.558117 17217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:12.558152 17217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:12.567659 17217 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:46767
I20260810 07:25:12.567690 17419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:46767 every 8 connection(s)
I20260810 07:25:12.568539 17217 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:25:12.571128 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 17217
I20260810 07:25:12.571270 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance
I20260810 07:25:12.571569 12919 external_mini_cluster.cc:1428] Setting key 272b0cabd2f9409c68f2069380427dfc
Aug 10 07:25:12 dist-test-slave-ffrd krb5kdc[16818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346712, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.157.195@KRBTEST.COM; defaulting to no policy
I20260810 07:25:12.584996 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:12.570349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:48455 (local address 127.12.157.254:37857)
0810 07:25:12.570510 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:12.570515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:12.571351 (+ 836us) server_negotiation.cc:407] Connection header received
0810 07:25:12.572881 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:12.572888 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:12.572973 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:12.573061 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:12.574244 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:12.575082 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:12.577033 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:12.577426 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:12.577557 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:12.580763 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:12.580770 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:12.580775 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:12.580819 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:12.582609 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:12.583167 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:12.583171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:12.583174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:12.583276 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:12.583642 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:12.583646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:12.583648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:12.583936 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:12.584055 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:12.584612 (+ 557us) server_negotiation.cc:299] Negotiation successful
0810 07:25:12.584816 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 07:25:12.585902 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:12.570725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37857 (local address 127.12.157.194:48455)
0810 07:25:12.571202 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:12.571255 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:12.572609 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:12.573249 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:12.573256 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:12.573617 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:12.574090 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:12.574103 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:12.575351 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:12.575356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:12.576853 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:12.576867 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:12.577586 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:12.577590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:12.577776 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:12.578534 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:12.578554 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:12.580574 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:12.582743 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:12.582748 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:12.582760 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:12.583048 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:12.583395 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:12.583398 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:12.583399 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:12.583539 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:12.584072 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:12.584078 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:12.584349 (+ 271us) client_negotiation.cc:769] Sending connection context
0810 07:25:12.584536 (+ 187us) client_negotiation.cc:240] Negotiation successful
0810 07:25:12.584967 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":145,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.195@KRBTEST.COM" created.
I20260810 07:25:12.587386 17420 heartbeater.cc:347] Connected to a master server at 127.12.157.254:37857
I20260810 07:25:12.587672 17420 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:12.588225 17420 heartbeater.cc:511] Master 127.12.157.254:37857 requested a full tablet report, sending...
I20260810 07:25:12.589447 16888 ts_manager.cc:194] Registered new tserver with Master: f3b623630c71429ca3610f7a10ee6815 (127.12.157.194:46767)
I20260810 07:25:12.590057 16888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.194@KRBTEST.COM'} at 127.12.157.194:48455
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
I20260810 07:25:12.604538 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:0
--local_ip_for_outbound_sockets=127.12.157.195
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37857
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:34301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:12.722044 17428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:12.722399 17428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:12.722486 17428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:12.726639 17428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:12.726718 17428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:12.726801 17428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:25:12.732064 17428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:34301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:12.733198 17428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:12.734202 17428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:12.743968 17436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:12.743973 17440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:12.744024 17438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:12.744210 17428 server_base.cc:1034] running on GCE node
I20260810 07:25:12.744786 17428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:12.745482 17428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:12.746680 17428 hybrid_clock.cc:648] HybridClock initialized: now 1786346712746650 us; error 44 us; skew 500 ppm
Aug 10 07:25:12 dist-test-slave-ffrd krb5kdc[16818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346712, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:12.749536 17428 init.cc:377] Logged in from keytab as kudu/127.12.157.195@KRBTEST.COM (short username kudu)
I20260810 07:25:12.750407 17428 webserver.cc:460] Webserver started at http://127.12.157.195:39695/ using document root <none> and password file <none>
I20260810 07:25:12.750918 17428 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:12.750977 17428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:12.751142 17428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:12.753182 17428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/data/instance:
uuid: "8e09d96ad0974539a87b076480b555b6"
format_stamp: "Formatted at 2026-08-10 07:25:12 on dist-test-slave-ffrd"
server_key: "f40ea53d0b957dc3cced06edc3b4d975"
server_key_iv: "6de80e3e8f1c711909264c663e729021"
server_key_version: "encryptionkey@0"
I20260810 07:25:12.753684 17428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance:
uuid: "8e09d96ad0974539a87b076480b555b6"
format_stamp: "Formatted at 2026-08-10 07:25:12 on dist-test-slave-ffrd"
server_key: "f40ea53d0b957dc3cced06edc3b4d975"
server_key_iv: "6de80e3e8f1c711909264c663e729021"
server_key_version: "encryptionkey@0"
I20260810 07:25:12.757300 17428 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 07:25:12.759600 17448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:12.760887 17428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 07:25:12.761085 17428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "8e09d96ad0974539a87b076480b555b6"
format_stamp: "Formatted at 2026-08-10 07:25:12 on dist-test-slave-ffrd"
server_key: "f40ea53d0b957dc3cced06edc3b4d975"
server_key_iv: "6de80e3e8f1c711909264c663e729021"
server_key_version: "encryptionkey@0"
I20260810 07:25:12.761188 17428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:12.782941 17428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:12.786121 17428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:12.786361 17428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:12.787024 17428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:12.787935 17428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:12.787985 17428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:12.788060 17428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:12.788097 17428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:12.798785 17428 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:45625
I20260810 07:25:12.798810 17639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:45625 every 8 connection(s)
I20260810 07:25:12.799824 17428 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:25:12.802330 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 17428
I20260810 07:25:12.802517 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance
I20260810 07:25:12.802886 12919 external_mini_cluster.cc:1428] Setting key de248f1721bf57e9e6c72cc7e99ef35f
Aug 10 07:25:12 dist-test-slave-ffrd krb5kdc[16818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346712, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:12.814141 16952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:12.801814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:48171 (local address 127.12.157.254:37857)
0810 07:25:12.801975 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:12.801980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:12.803270 (+ 1290us) server_negotiation.cc:407] Connection header received
0810 07:25:12.804321 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:12.804325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:12.804390 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:12.804499 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:12.805844 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:12.806394 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:12.807574 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:12.807797 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:12.807913 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:12.810580 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:12.810591 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:12.810593 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:12.810617 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:12.811870 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:12.812388 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:12.812397 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:12.812399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:12.812475 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:12.812828 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:12.812833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:12.812834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:12.813075 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:12.813186 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:12.813830 (+ 644us) server_negotiation.cc:299] Negotiation successful
0810 07:25:12.813975 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 07:25:12.814965 17643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:12.802252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37857 (local address 127.12.157.195:48171)
0810 07:25:12.803088 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:12.803148 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:12.804098 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:12.804685 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:12.804692 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:12.805164 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:12.805702 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:12.805716 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:12.806553 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:12.806561 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:12.807446 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:12.807456 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:12.807958 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:12.807962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:12.808175 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:12.808796 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:12.808813 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:12.810417 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:12.811995 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:12.812001 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:12.812015 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:12.812270 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:12.812572 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:12.812576 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:12.812578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:12.812716 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:12.813185 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:12.813192 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:12.813550 (+ 358us) client_negotiation.cc:769] Sending connection context
0810 07:25:12.813783 (+ 233us) client_negotiation.cc:240] Negotiation successful
0810 07:25:12.814159 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":146,"threads_started":1}
I20260810 07:25:12.816265 17640 heartbeater.cc:347] Connected to a master server at 127.12.157.254:37857
I20260810 07:25:12.816524 17640 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:12.817163 17640 heartbeater.cc:511] Master 127.12.157.254:37857 requested a full tablet report, sending...
I20260810 07:25:12.818194 16888 ts_manager.cc:194] Registered new tserver with Master: 8e09d96ad0974539a87b076480b555b6 (127.12.157.195:45625)
I20260810 07:25:12.818730 16888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.195@KRBTEST.COM'} at 127.12.157.195:48171
I20260810 07:25:12.828617 12919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 07:25:12.844060 16952 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:12.842596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35962 (local address 127.12.157.254:37857)
0810 07:25:12.842753 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:12.842757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:12.842894 (+ 137us) server_negotiation.cc:407] Connection header received
0810 07:25:12.843007 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:12.843010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:12.843078 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:12.843214 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:12.843950 (+ 736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35962: BlockingRecv error: recv got EOF from 127.0.0.1:35962 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 07:25:12.845197 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 16990
I20260810 07:25:12.867650 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 17217
I20260810 07:25:12.879894 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 17428
I20260810 07:25:12.889550 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 16838
2026-08-10T07:25:12Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2476 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:25:12 dist-test-slave-ffrd krb5kdc[17665](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:25:12 dist-test-slave-ffrd krb5kdc[17665](info): set up 2 sockets
Aug 10 07:25:12 dist-test-slave-ffrd krb5kdc[17665](info): commencing operation
krb5kdc: starting...
W20260810 07:25:14.171150 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.252s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:25:14 dist-test-slave-ffrd krb5kdc[17665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:25:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:25:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:25:14.304136 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:39887
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:43563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:39887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:14.419912 17689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:14.420279 17689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:14.420357 17689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:14.424579 17689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:25:14.424700 17689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:14.424746 17689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:14.424785 17689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:25:14.424825 17689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:25:14.430424 17689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:43563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:39887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:39887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:14.431614 17689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:14.432703 17689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:14.441875 17694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:14.441890 17699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:14.441875 17702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:14.441947 17689 server_base.cc:1034] running on GCE node
I20260810 07:25:14.442843 17689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:14.444067 17689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:14.445264 17689 hybrid_clock.cc:648] HybridClock initialized: now 1786346714445233 us; error 46 us; skew 500 ppm
Aug 10 07:25:14 dist-test-slave-ffrd krb5kdc[17665](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346714, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:14.448313 17689 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:25:14.449337 17689 webserver.cc:460] Webserver started at http://127.12.157.254:35353/ using document root <none> and password file <none>
I20260810 07:25:14.449966 17689 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:14.450070 17689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:14.450330 17689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:14.452605 17689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "8a3226624b3e4f1a9b57b6f6bf9eeb32"
format_stamp: "Formatted at 2026-08-10 07:25:14 on dist-test-slave-ffrd"
server_key: "2fadfc3406fb137abd2f63a271551bdf"
server_key_iv: "9482e36f86c8cf2edb048babddca61ca"
server_key_version: "encryptionkey@0"
I20260810 07:25:14.453245 17689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "8a3226624b3e4f1a9b57b6f6bf9eeb32"
format_stamp: "Formatted at 2026-08-10 07:25:14 on dist-test-slave-ffrd"
server_key: "2fadfc3406fb137abd2f63a271551bdf"
server_key_iv: "9482e36f86c8cf2edb048babddca61ca"
server_key_version: "encryptionkey@0"
I20260810 07:25:14.457455 17689 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 07:25:14.460412 17712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:14.461935 17689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:25:14.462105 17689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "8a3226624b3e4f1a9b57b6f6bf9eeb32"
format_stamp: "Formatted at 2026-08-10 07:25:14 on dist-test-slave-ffrd"
server_key: "2fadfc3406fb137abd2f63a271551bdf"
server_key_iv: "9482e36f86c8cf2edb048babddca61ca"
server_key_version: "encryptionkey@0"
I20260810 07:25:14.462239 17689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:14.476871 17689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:14.480324 17689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:14.480500 17689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:14.488083 17689 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:39887
I20260810 07:25:14.488157 17801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:39887 every 8 connection(s)
I20260810 07:25:14.489192 17689 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:25:14.491824 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 17689
I20260810 07:25:14.491981 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:25:14.492265 12919 external_mini_cluster.cc:1428] Setting key 0587d61e2cd13950970549885b7f31f5
I20260810 07:25:14.492380 17803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:14.499655 17803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a3226624b3e4f1a9b57b6f6bf9eeb32: Bootstrap starting.
Aug 10 07:25:14 dist-test-slave-ffrd krb5kdc[17665](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:14.502002 17803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a3226624b3e4f1a9b57b6f6bf9eeb32: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:14.502741 17803 log.cc:824] T 00000000000000000000000000000000 P 8a3226624b3e4f1a9b57b6f6bf9eeb32: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:14.504659 17803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a3226624b3e4f1a9b57b6f6bf9eeb32: No bootstrap required, opened a new log
I20260810 07:25:14.507115 17809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:14.493758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45536 (local address 127.12.157.254:39887)
0810 07:25:14.494208 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:14.494225 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:14.494261 (+ 36us) server_negotiation.cc:407] Connection header received
0810 07:25:14.495206 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:14.495228 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:14.495599 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:14.495903 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:14.496912 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:14.497991 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:14.499190 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:14.499679 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:14.499937 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:14.502547 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:14.502552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:14.502565 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:14.502594 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:14.504308 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:14.504795 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:14.504800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:14.504804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:14.504917 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:14.505263 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:14.505267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:14.505270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:14.505663 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:14.505888 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:14.506121 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 07:25:14.506415 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":167,"threads_started":1}
I20260810 07:25:14.508656 17803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a3226624b3e4f1a9b57b6f6bf9eeb32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3226624b3e4f1a9b57b6f6bf9eeb32" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39887 } }
I20260810 07:25:14.508905 17803 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a3226624b3e4f1a9b57b6f6bf9eeb32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:14.509032 17803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a3226624b3e4f1a9b57b6f6bf9eeb32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a3226624b3e4f1a9b57b6f6bf9eeb32, State: Initialized, Role: FOLLOWER
I20260810 07:25:14.509488 17803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a3226624b3e4f1a9b57b6f6bf9eeb32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3226624b3e4f1a9b57b6f6bf9eeb32" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39887 } }
I20260810 07:25:14.509649 17803 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a3226624b3e4f1a9b57b6f6bf9eeb32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:25:14.509722 17803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a3226624b3e4f1a9b57b6f6bf9eeb32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:25:14.509846 17803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a3226624b3e4f1a9b57b6f6bf9eeb32 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:14.510272 12919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:25:14.510895 17803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a3226624b3e4f1a9b57b6f6bf9eeb32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3226624b3e4f1a9b57b6f6bf9eeb32" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39887 } }
I20260810 07:25:14.511233 17803 leader_election.cc:302] T 00000000000000000000000000000000 P 8a3226624b3e4f1a9b57b6f6bf9eeb32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a3226624b3e4f1a9b57b6f6bf9eeb32; no voters:
I20260810 07:25:14.511572 17803 leader_election.cc:288] T 00000000000000000000000000000000 P 8a3226624b3e4f1a9b57b6f6bf9eeb32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:25:14.511686 17812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a3226624b3e4f1a9b57b6f6bf9eeb32 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:14.511925 17812 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a3226624b3e4f1a9b57b6f6bf9eeb32 [term 1 LEADER]: Becoming Leader. State: Replica: 8a3226624b3e4f1a9b57b6f6bf9eeb32, State: Running, Role: LEADER
I20260810 07:25:14.512279 17812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a3226624b3e4f1a9b57b6f6bf9eeb32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3226624b3e4f1a9b57b6f6bf9eeb32" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39887 } }
I20260810 07:25:14.512920 17803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a3226624b3e4f1a9b57b6f6bf9eeb32 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:25:14.514014 17815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a3226624b3e4f1a9b57b6f6bf9eeb32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a3226624b3e4f1a9b57b6f6bf9eeb32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3226624b3e4f1a9b57b6f6bf9eeb32" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39887 } } }
I20260810 07:25:14.514146 17815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a3226624b3e4f1a9b57b6f6bf9eeb32 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:14.514456 17816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a3226624b3e4f1a9b57b6f6bf9eeb32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a3226624b3e4f1a9b57b6f6bf9eeb32. Latest consensus state: current_term: 1 leader_uuid: "8a3226624b3e4f1a9b57b6f6bf9eeb32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a3226624b3e4f1a9b57b6f6bf9eeb32" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39887 } } }
I20260810 07:25:14.514546 17816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a3226624b3e4f1a9b57b6f6bf9eeb32 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:14.514928 17828 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:25:14.518711 17828 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:25:14.524868 17828 catalog_manager.cc:1347] Generated new cluster ID: a934b6f9e2e341a6aa8ba80ae8c25aba
I20260810 07:25:14.524928 17828 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:25:14.532189 17809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:14.526462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45550 (local address 127.12.157.254:39887)
0810 07:25:14.526577 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:14.526582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:14.526970 (+ 388us) server_negotiation.cc:407] Connection header received
0810 07:25:14.527085 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:14.527088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:14.527145 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:14.527231 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:14.527793 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:14.528620 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:14.529766 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:14.530152 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:14.530263 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:14.531171 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:14.531175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:14.531178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:25:14.531728 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:14.531763 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:25:14.531782 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:14.531875 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:14.531983 (+ 108us) server_negotiation.cc:299] Negotiation successful
0810 07:25:14.532045 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 07:25:14.545624 17828 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:25:14.546602 17828 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:25:14.561834 17828 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a3226624b3e4f1a9b57b6f6bf9eeb32: Generated new TSK 0
I20260810 07:25:14.562626 17828 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:25:14.574175 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 17689
2026-08-10T07:25:14Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1679 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:25:14 dist-test-slave-ffrd krb5kdc[17854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:25:14 dist-test-slave-ffrd krb5kdc[17854](info): set up 2 sockets
Aug 10 07:25:14 dist-test-slave-ffrd krb5kdc[17854](info): commencing operation
krb5kdc: starting...
W20260810 07:25:15.857024 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.259s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:25:15 dist-test-slave-ffrd krb5kdc[17854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:25:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:25:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:25:15.989972 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:45907
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:35291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:45907
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:16.114506 17872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:16.114851 17872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:16.114928 17872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:16.119136 17872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:25:16.119239 17872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:16.119275 17872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:16.119313 17872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:25:16.119345 17872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:25:16.125229 17872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:35291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:45907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:45907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:16.126427 17872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:16.127516 17872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:16.136615 17888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:16.136718 17872 server_base.cc:1034] running on GCE node
W20260810 07:25:16.136801 17891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:16.136729 17893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:16.137625 17872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:16.138705 17872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:16.139873 17872 hybrid_clock.cc:648] HybridClock initialized: now 1786346716139844 us; error 45 us; skew 500 ppm
Aug 10 07:25:16 dist-test-slave-ffrd krb5kdc[17854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346716, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:16.149590 17872 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:25:16.150595 17872 webserver.cc:460] Webserver started at http://127.12.157.254:38501/ using document root <none> and password file <none>
I20260810 07:25:16.151203 17872 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:16.151268 17872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:16.151518 17872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:16.153700 17872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "12a21a2900cd42e9ae565990b3283bcc"
format_stamp: "Formatted at 2026-08-10 07:25:16 on dist-test-slave-ffrd"
server_key: "c847a7a0cfbcc454a1413516695bbb8b"
server_key_iv: "5ce9f499274f3b0ddcff04fa6fe55b96"
server_key_version: "encryptionkey@0"
I20260810 07:25:16.154294 17872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "12a21a2900cd42e9ae565990b3283bcc"
format_stamp: "Formatted at 2026-08-10 07:25:16 on dist-test-slave-ffrd"
server_key: "c847a7a0cfbcc454a1413516695bbb8b"
server_key_iv: "5ce9f499274f3b0ddcff04fa6fe55b96"
server_key_version: "encryptionkey@0"
I20260810 07:25:16.158195 17872 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 07:25:16.160660 17902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:16.162166 17872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 07:25:16.162328 17872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "12a21a2900cd42e9ae565990b3283bcc"
format_stamp: "Formatted at 2026-08-10 07:25:16 on dist-test-slave-ffrd"
server_key: "c847a7a0cfbcc454a1413516695bbb8b"
server_key_iv: "5ce9f499274f3b0ddcff04fa6fe55b96"
server_key_version: "encryptionkey@0"
I20260810 07:25:16.162461 17872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:16.177515 17872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:16.180441 17872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:16.180656 17872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:16.187855 17872 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:45907
I20260810 07:25:16.188148 17982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:45907 every 8 connection(s)
I20260810 07:25:16.188926 17872 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:25:16.191871 17985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:16.197564 17985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12a21a2900cd42e9ae565990b3283bcc: Bootstrap starting.
I20260810 07:25:16.198480 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 17872
I20260810 07:25:16.198635 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:25:16.198969 12919 external_mini_cluster.cc:1428] Setting key e26d8d8ae596ee7e8b6b1f3c437191a1
I20260810 07:25:16.200593 17985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12a21a2900cd42e9ae565990b3283bcc: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:16.201408 17985 log.cc:824] T 00000000000000000000000000000000 P 12a21a2900cd42e9ae565990b3283bcc: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:16.203244 17985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12a21a2900cd42e9ae565990b3283bcc: No bootstrap required, opened a new log
I20260810 07:25:16.206938 17985 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12a21a2900cd42e9ae565990b3283bcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a21a2900cd42e9ae565990b3283bcc" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 45907 } }
I20260810 07:25:16.207187 17985 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12a21a2900cd42e9ae565990b3283bcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:16.207280 17985 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12a21a2900cd42e9ae565990b3283bcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a21a2900cd42e9ae565990b3283bcc, State: Initialized, Role: FOLLOWER
I20260810 07:25:16.207764 17985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12a21a2900cd42e9ae565990b3283bcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a21a2900cd42e9ae565990b3283bcc" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 45907 } }
Aug 10 07:25:16 dist-test-slave-ffrd krb5kdc[17854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:16.207919 17985 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12a21a2900cd42e9ae565990b3283bcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:25:16.208005 17985 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12a21a2900cd42e9ae565990b3283bcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:25:16.208104 17985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12a21a2900cd42e9ae565990b3283bcc [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:16.209267 17985 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12a21a2900cd42e9ae565990b3283bcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a21a2900cd42e9ae565990b3283bcc" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 45907 } }
I20260810 07:25:16.209663 17985 leader_election.cc:302] T 00000000000000000000000000000000 P 12a21a2900cd42e9ae565990b3283bcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12a21a2900cd42e9ae565990b3283bcc; no voters:
I20260810 07:25:16.210047 17985 leader_election.cc:288] T 00000000000000000000000000000000 P 12a21a2900cd42e9ae565990b3283bcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:25:16.210111 17991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12a21a2900cd42e9ae565990b3283bcc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:16.210315 17991 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12a21a2900cd42e9ae565990b3283bcc [term 1 LEADER]: Becoming Leader. State: Replica: 12a21a2900cd42e9ae565990b3283bcc, State: Running, Role: LEADER
I20260810 07:25:16.210659 17991 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12a21a2900cd42e9ae565990b3283bcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a21a2900cd42e9ae565990b3283bcc" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 45907 } }
I20260810 07:25:16.211362 17985 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12a21a2900cd42e9ae565990b3283bcc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:25:16.212397 17992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12a21a2900cd42e9ae565990b3283bcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12a21a2900cd42e9ae565990b3283bcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a21a2900cd42e9ae565990b3283bcc" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 45907 } } }
I20260810 07:25:16.212543 17992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12a21a2900cd42e9ae565990b3283bcc [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:16.212917 17993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12a21a2900cd42e9ae565990b3283bcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12a21a2900cd42e9ae565990b3283bcc. Latest consensus state: current_term: 1 leader_uuid: "12a21a2900cd42e9ae565990b3283bcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12a21a2900cd42e9ae565990b3283bcc" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 45907 } } }
I20260810 07:25:16.213037 17993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12a21a2900cd42e9ae565990b3283bcc [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:16.214074 18004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:25:16.214658 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:16.200131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43730 (local address 127.12.157.254:45907)
0810 07:25:16.200756 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:16.200774 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:16.200861 (+ 87us) server_negotiation.cc:407] Connection header received
0810 07:25:16.201640 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:16.201659 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:16.201988 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:16.202321 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:16.203150 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:16.204197 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:16.205114 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:16.205604 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:16.205791 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:16.208568 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:16.208573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:16.208587 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:16.208615 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:16.210546 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:16.211046 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:16.211050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:16.211055 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:16.211160 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:16.211464 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:16.211468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:16.211470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:16.212007 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:16.212979 (+ 972us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:16.213134 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 07:25:16.213479 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":165,"threads_started":1}
I20260810 07:25:16.217334 18004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:25:16.222826 18004 catalog_manager.cc:1347] Generated new cluster ID: 07d85f58df684ae18b31e2e29ce508aa
I20260810 07:25:16.222899 18004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:25:16.230155 18004 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:25:16.231072 18004 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:25:16.237792 18004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12a21a2900cd42e9ae565990b3283bcc: Generated new TSK 0
I20260810 07:25:16.238610 18004 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
I20260810 07:25:16.272424 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:0
--local_ip_for_outbound_sockets=127.12.157.193
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:45907
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:35291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:16.386574 18026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:16.386834 18026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:16.386878 18026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:16.390950 18026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:16.391006 18026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:16.391098 18026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:25:16.396320 18026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:35291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:45907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:16.397395 18026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:16.398412 18026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:16.409008 18038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:16.409032 18044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:16.409302 18026 server_base.cc:1034] running on GCE node
W20260810 07:25:16.409036 18042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:16.409946 18026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:16.410677 18026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:16.411926 18026 hybrid_clock.cc:648] HybridClock initialized: now 1786346716411909 us; error 51 us; skew 500 ppm
Aug 10 07:25:16 dist-test-slave-ffrd krb5kdc[17854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346716, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:16.418133 18026 init.cc:377] Logged in from keytab as kudu/127.12.157.193@KRBTEST.COM (short username kudu)
I20260810 07:25:16.419337 18026 webserver.cc:460] Webserver started at http://127.12.157.193:36083/ using document root <none> and password file <none>
I20260810 07:25:16.419965 18026 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:16.420028 18026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:16.420261 18026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:16.422483 18026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/data/instance:
uuid: "117797ed49fe4f1cb359fd2b097aedf8"
format_stamp: "Formatted at 2026-08-10 07:25:16 on dist-test-slave-ffrd"
server_key: "7dced426aeda6724dad84c323862839b"
server_key_iv: "34eed1f3e5f3cb88f5210ab76408b842"
server_key_version: "encryptionkey@0"
I20260810 07:25:16.423133 18026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance:
uuid: "117797ed49fe4f1cb359fd2b097aedf8"
format_stamp: "Formatted at 2026-08-10 07:25:16 on dist-test-slave-ffrd"
server_key: "7dced426aeda6724dad84c323862839b"
server_key_iv: "34eed1f3e5f3cb88f5210ab76408b842"
server_key_version: "encryptionkey@0"
I20260810 07:25:16.427196 18026 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260810 07:25:16.430172 18055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:16.431520 18026 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 07:25:16.431641 18026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "117797ed49fe4f1cb359fd2b097aedf8"
format_stamp: "Formatted at 2026-08-10 07:25:16 on dist-test-slave-ffrd"
server_key: "7dced426aeda6724dad84c323862839b"
server_key_iv: "34eed1f3e5f3cb88f5210ab76408b842"
server_key_version: "encryptionkey@0"
I20260810 07:25:16.431737 18026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:16.468750 18026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:16.472361 18026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:16.472528 18026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:16.473145 18026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:16.474036 18026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:16.474113 18026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:16.474188 18026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:16.474233 18026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:16.483752 18026 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:36917
I20260810 07:25:16.483775 18238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:36917 every 8 connection(s)
I20260810 07:25:16.484702 18026 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:25:16.490525 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 18026
I20260810 07:25:16.490684 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance
I20260810 07:25:16.490980 12919 external_mini_cluster.cc:1428] Setting key 57e4fe0c84f04d0ef0f266181248a9b1
Aug 10 07:25:16 dist-test-slave-ffrd krb5kdc[17854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346716, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:16.500880 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:16.486718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:36467 (local address 127.12.157.254:45907)
0810 07:25:16.486898 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:16.486903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:16.487753 (+ 850us) server_negotiation.cc:407] Connection header received
0810 07:25:16.489151 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:16.489154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:16.489204 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:16.489281 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:16.490604 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:16.491296 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:16.492919 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:16.493224 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:16.493309 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:16.496306 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:16.496313 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:16.496319 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:16.496516 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:16.498460 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:16.499042 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:16.499049 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:16.499053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:16.499149 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:16.499522 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:16.499527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:16.499532 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:16.499818 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:16.499955 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:16.500500 (+ 545us) server_negotiation.cc:299] Negotiation successful
0810 07:25:16.500644 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 07:25:16.501817 18245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:16.487105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:45907 (local address 127.12.157.193:36467)
0810 07:25:16.487589 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:16.487633 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:16.488882 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:16.489530 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:16.489537 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:16.489933 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:16.490428 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:16.490441 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:16.491470 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:16.491473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:16.492783 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:16.492794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:16.493394 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:16.493400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:16.493653 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:16.494347 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:16.494377 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:16.496126 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:16.498621 (+ 2495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:16.498627 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:16.498640 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:16.498924 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:16.499255 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:16.499258 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:16.499259 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:16.499415 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:16.499969 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:16.499975 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:16.500232 (+ 257us) client_negotiation.cc:769] Sending connection context
0810 07:25:16.500416 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 07:25:16.500782 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for kudu/127.12.157.194@KRBTEST.COM; defaulting to no policy
I20260810 07:25:16.503218 18239 heartbeater.cc:347] Connected to a master server at 127.12.157.254:45907
I20260810 07:25:16.503492 18239 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:16.504169 18239 heartbeater.cc:511] Master 127.12.157.254:45907 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.194@KRBTEST.COM" created.
I20260810 07:25:16.506244 17930 ts_manager.cc:194] Registered new tserver with Master: 117797ed49fe4f1cb359fd2b097aedf8 (127.12.157.193:36917)
I20260810 07:25:16.508306 17930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.193@KRBTEST.COM'} at 127.12.157.193:36467
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
I20260810 07:25:16.529091 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:0
--local_ip_for_outbound_sockets=127.12.157.194
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:45907
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:35291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:16.651552 18250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:16.651819 18250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:16.651863 18250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:16.655984 18250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:16.656047 18250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:16.656140 18250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:25:16.661623 18250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:35291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:45907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:16.662717 18250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:16.663789 18250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:16.673622 18260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:16.673666 18256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:16.673682 18257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:16.673851 18250 server_base.cc:1034] running on GCE node
I20260810 07:25:16.674278 18250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:16.674925 18250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:16.676118 18250 hybrid_clock.cc:648] HybridClock initialized: now 1786346716676102 us; error 26 us; skew 500 ppm
Aug 10 07:25:16 dist-test-slave-ffrd krb5kdc[17854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346716, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:16.679023 18250 init.cc:377] Logged in from keytab as kudu/127.12.157.194@KRBTEST.COM (short username kudu)
I20260810 07:25:16.679880 18250 webserver.cc:460] Webserver started at http://127.12.157.194:37247/ using document root <none> and password file <none>
I20260810 07:25:16.680442 18250 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:16.680536 18250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:16.680770 18250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:16.682848 18250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/data/instance:
uuid: "994b96b94fb5472e8e25b424d0f8a961"
format_stamp: "Formatted at 2026-08-10 07:25:16 on dist-test-slave-ffrd"
server_key: "9615090bec275326fba282406cbc5047"
server_key_iv: "22cd0bfcdceb254f99b23b8d5b021ce2"
server_key_version: "encryptionkey@0"
I20260810 07:25:16.683450 18250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance:
uuid: "994b96b94fb5472e8e25b424d0f8a961"
format_stamp: "Formatted at 2026-08-10 07:25:16 on dist-test-slave-ffrd"
server_key: "9615090bec275326fba282406cbc5047"
server_key_iv: "22cd0bfcdceb254f99b23b8d5b021ce2"
server_key_version: "encryptionkey@0"
I20260810 07:25:16.687057 18250 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260810 07:25:16.689399 18270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:16.690873 18250 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:25:16.690995 18250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "994b96b94fb5472e8e25b424d0f8a961"
format_stamp: "Formatted at 2026-08-10 07:25:16 on dist-test-slave-ffrd"
server_key: "9615090bec275326fba282406cbc5047"
server_key_iv: "22cd0bfcdceb254f99b23b8d5b021ce2"
server_key_version: "encryptionkey@0"
I20260810 07:25:16.691115 18250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:16.721290 18250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:16.724581 18250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:16.724793 18250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:16.725494 18250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:16.726321 18250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:16.726401 18250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:16.726495 18250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:16.726544 18250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:16.735647 18250 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:45039
I20260810 07:25:16.735662 18465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:45039 every 8 connection(s)
I20260810 07:25:16.736713 18250 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:25:16.737845 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 18250
I20260810 07:25:16.737972 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance
I20260810 07:25:16.738304 12919 external_mini_cluster.cc:1428] Setting key bc3f2321c60d790cd188a86a46967a6d
Aug 10 07:25:16 dist-test-slave-ffrd krb5kdc[17854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346716, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.195@KRBTEST.COM" created.
I20260810 07:25:16.751874 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:16.738808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:33569 (local address 127.12.157.254:45907)
0810 07:25:16.738943 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:16.738947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:16.740029 (+ 1082us) server_negotiation.cc:407] Connection header received
0810 07:25:16.742095 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:16.742100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:16.742164 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:16.742271 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:16.743421 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:16.744004 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:16.745216 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:16.745439 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:16.745544 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:16.748230 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:16.748233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:16.748235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:16.748257 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:16.749818 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:16.750347 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:16.750351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:16.750353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:16.750445 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:16.750789 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:16.750793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:16.750795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:16.750984 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:16.751094 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:16.751588 (+ 494us) server_negotiation.cc:299] Negotiation successful
0810 07:25:16.751711 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 07:25:16.752759 18469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:16.739292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:45907 (local address 127.12.157.194:33569)
0810 07:25:16.739864 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:16.739924 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:16.741861 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:16.742460 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:16.742466 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:16.742806 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:16.743276 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:16.743289 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:16.744134 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:16.744138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:16.745092 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:16.745102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:16.745597 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:16.745601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:16.745774 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:16.746354 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:16.746378 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:16.748055 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:16.749961 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:16.749967 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:16.749982 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:16.750241 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:16.750541 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:16.750544 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:16.750546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:16.750690 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:16.751094 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:16.751102 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:16.751354 (+ 252us) client_negotiation.cc:769] Sending connection context
0810 07:25:16.751546 (+ 192us) client_negotiation.cc:240] Negotiation successful
0810 07:25:16.751882 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":130,"threads_started":1}
I20260810 07:25:16.754045 18466 heartbeater.cc:347] Connected to a master server at 127.12.157.254:45907
I20260810 07:25:16.754298 18466 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:16.754794 18466 heartbeater.cc:511] Master 127.12.157.254:45907 requested a full tablet report, sending...
I20260810 07:25:16.755857 17930 ts_manager.cc:194] Registered new tserver with Master: 994b96b94fb5472e8e25b424d0f8a961 (127.12.157.194:45039)
I20260810 07:25:16.756518 17930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.194@KRBTEST.COM'} at 127.12.157.194:33569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
I20260810 07:25:16.768916 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:0
--local_ip_for_outbound_sockets=127.12.157.195
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:45907
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:35291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:16.886940 18472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:16.887311 18472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:16.887409 18472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:16.892627 18472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:16.892751 18472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:16.892890 18472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:25:16.899209 18472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:35291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:45907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:16.900408 18472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:16.901646 18472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:16.912168 18478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:16.912180 18477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:16.912421 18472 server_base.cc:1034] running on GCE node
W20260810 07:25:16.912168 18481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:16.913048 18472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:16.913736 18472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:16.914942 18472 hybrid_clock.cc:648] HybridClock initialized: now 1786346716914907 us; error 47 us; skew 500 ppm
Aug 10 07:25:16 dist-test-slave-ffrd krb5kdc[17854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346716, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:16.918581 18472 init.cc:377] Logged in from keytab as kudu/127.12.157.195@KRBTEST.COM (short username kudu)
I20260810 07:25:16.919546 18472 webserver.cc:460] Webserver started at http://127.12.157.195:38619/ using document root <none> and password file <none>
I20260810 07:25:16.920148 18472 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:16.920243 18472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:16.920483 18472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:16.922688 18472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/data/instance:
uuid: "69d7c778510841ae88e1f8889ff02484"
format_stamp: "Formatted at 2026-08-10 07:25:16 on dist-test-slave-ffrd"
server_key: "421e8d8ef2a50f898f7fae62c44cfe06"
server_key_iv: "018bb4306ab3201a1538708b576d71a7"
server_key_version: "encryptionkey@0"
I20260810 07:25:16.923283 18472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance:
uuid: "69d7c778510841ae88e1f8889ff02484"
format_stamp: "Formatted at 2026-08-10 07:25:16 on dist-test-slave-ffrd"
server_key: "421e8d8ef2a50f898f7fae62c44cfe06"
server_key_iv: "018bb4306ab3201a1538708b576d71a7"
server_key_version: "encryptionkey@0"
I20260810 07:25:16.927219 18472 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260810 07:25:16.930007 18490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:16.931394 18472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:25:16.931565 18472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "69d7c778510841ae88e1f8889ff02484"
format_stamp: "Formatted at 2026-08-10 07:25:16 on dist-test-slave-ffrd"
server_key: "421e8d8ef2a50f898f7fae62c44cfe06"
server_key_iv: "018bb4306ab3201a1538708b576d71a7"
server_key_version: "encryptionkey@0"
I20260810 07:25:16.931700 18472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:16.963572 18472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:16.967029 18472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:16.967245 18472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:16.967941 18472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:16.968815 18472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:16.968888 18472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:16.969009 18472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:16.969053 18472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:16.978456 18472 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:41783
I20260810 07:25:16.978511 18680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:41783 every 8 connection(s)
I20260810 07:25:16.979434 18472 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:25:16.987071 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 18472
I20260810 07:25:16.987241 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance
I20260810 07:25:16.987613 12919 external_mini_cluster.cc:1428] Setting key 6834a7a4d88f25a3a5558448ee66d42c
Aug 10 07:25:16 dist-test-slave-ffrd krb5kdc[17854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346716, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:16.994307 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:16.981495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:32969 (local address 127.12.157.254:45907)
0810 07:25:16.981662 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:16.981667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:16.982572 (+ 905us) server_negotiation.cc:407] Connection header received
0810 07:25:16.983902 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:16.983905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:16.983958 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:16.984037 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:16.985476 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:16.986138 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:16.987489 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:16.987832 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:16.987962 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:16.990903 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:16.990906 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:16.990908 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:16.990934 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:16.992308 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:16.992832 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:16.992835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:16.992836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:16.992915 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:16.993324 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:16.993326 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:16.993328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:16.993514 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:16.993593 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:16.994089 (+ 496us) server_negotiation.cc:299] Negotiation successful
0810 07:25:16.994188 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 07:25:16.995177 18685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:16.981934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:45907 (local address 127.12.157.195:32969)
0810 07:25:16.982394 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:16.982451 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:16.983694 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:16.984261 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:16.984269 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:16.984692 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:16.985313 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:16.985328 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:16.986294 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:16.986297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:16.987341 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:16.987361 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:16.988033 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:16.988038 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:16.988261 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:16.989005 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:16.989031 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:16.990723 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:16.992426 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:16.992432 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:16.992444 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:16.992700 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:16.993090 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:16.993093 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:16.993094 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:16.993232 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:16.993614 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:16.993620 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:16.993867 (+ 247us) client_negotiation.cc:769] Sending connection context
0810 07:25:16.994037 (+ 170us) client_negotiation.cc:240] Negotiation successful
0810 07:25:16.994318 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":120,"threads_started":1}
I20260810 07:25:16.996424 18681 heartbeater.cc:347] Connected to a master server at 127.12.157.254:45907
I20260810 07:25:16.996634 18681 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:16.997159 18681 heartbeater.cc:511] Master 127.12.157.254:45907 requested a full tablet report, sending...
I20260810 07:25:16.998320 17930 ts_manager.cc:194] Registered new tserver with Master: 69d7c778510841ae88e1f8889ff02484 (127.12.157.195:41783)
I20260810 07:25:16.998878 17930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.195@KRBTEST.COM'} at 127.12.157.195:32969
I20260810 07:25:17.001914 12919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 07:25:17 dist-test-slave-ffrd krb5kdc[17854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346717, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 07:25:17 dist-test-slave-ffrd krb5kdc[17854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346717, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:17.031641 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:17.022431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43734 (local address 127.12.157.254:45907)
0810 07:25:17.022629 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:17.022634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:17.022777 (+ 143us) server_negotiation.cc:407] Connection header received
0810 07:25:17.022980 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:17.022983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:17.023029 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:17.023118 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:17.023884 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:17.024442 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:17.025425 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:17.025646 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:17.025759 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:17.028302 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:17.028305 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:17.028307 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:17.028334 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:17.030029 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:17.030482 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:17.030485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:17.030486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:17.030572 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:17.030822 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:17.030824 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:17.030826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:17.031069 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:17.031176 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:17.031405 (+ 229us) server_negotiation.cc:299] Negotiation successful
0810 07:25:17.031515 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260810 07:25:17.032099 17930 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43734
I20260810 07:25:17.033265 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 18026
I20260810 07:25:17.048580 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 18250
I20260810 07:25:17.059938 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 18472
I20260810 07:25:17.076694 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 17872
2026-08-10T07:25:17Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2515 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:25:17 dist-test-slave-ffrd krb5kdc[18704](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:25:17 dist-test-slave-ffrd krb5kdc[18704](info): set up 2 sockets
Aug 10 07:25:17 dist-test-slave-ffrd krb5kdc[18704](info): commencing operation
krb5kdc: starting...
W20260810 07:25:18.379989 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.266s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:25:18 dist-test-slave-ffrd krb5kdc[18704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:25:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:25:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:25:18.516680 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:46121
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:39679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:46121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:18.634105 18727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:18.634446 18727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:18.634542 18727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:18.638774 18727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:25:18.638880 18727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:18.638917 18727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:18.638962 18727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:25:18.639014 18727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:25:18.644659 18727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:39679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:46121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:46121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:18.645887 18727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:18.646991 18727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 07:25:18.657133 18727 server_base.cc:1034] running on GCE node
W20260810 07:25:18.657147 18742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:18.657167 18732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:18.657493 18738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:18.658180 18727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:18.659416 18727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:18.660624 18727 hybrid_clock.cc:648] HybridClock initialized: now 1786346718660598 us; error 43 us; skew 500 ppm
Aug 10 07:25:18 dist-test-slave-ffrd krb5kdc[18704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346718, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:18.663856 18727 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:25:18.664851 18727 webserver.cc:460] Webserver started at http://127.12.157.254:37357/ using document root <none> and password file <none>
I20260810 07:25:18.665469 18727 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:18.665562 18727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:18.665807 18727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:18.667901 18727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "d657f1f0a465440a93702ced1b22f292"
format_stamp: "Formatted at 2026-08-10 07:25:18 on dist-test-slave-ffrd"
server_key: "d00c5d3a7ecda7641b34e48c85bc0c52"
server_key_iv: "8d3959d4f31be0ece99cb94cc54220c6"
server_key_version: "encryptionkey@0"
I20260810 07:25:18.668581 18727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "d657f1f0a465440a93702ced1b22f292"
format_stamp: "Formatted at 2026-08-10 07:25:18 on dist-test-slave-ffrd"
server_key: "d00c5d3a7ecda7641b34e48c85bc0c52"
server_key_iv: "8d3959d4f31be0ece99cb94cc54220c6"
server_key_version: "encryptionkey@0"
I20260810 07:25:18.672446 18727 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 07:25:18.674881 18755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:18.676174 18727 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 07:25:18.676323 18727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "d657f1f0a465440a93702ced1b22f292"
format_stamp: "Formatted at 2026-08-10 07:25:18 on dist-test-slave-ffrd"
server_key: "d00c5d3a7ecda7641b34e48c85bc0c52"
server_key_iv: "8d3959d4f31be0ece99cb94cc54220c6"
server_key_version: "encryptionkey@0"
I20260810 07:25:18.676436 18727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:18.690204 18727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:18.693459 18727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:18.693665 18727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:18.701139 18727 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:46121
I20260810 07:25:18.701167 18833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:46121 every 8 connection(s)
I20260810 07:25:18.702091 18727 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:25:18.704842 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 18727
I20260810 07:25:18.705042 18834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:18.705054 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:25:18.705377 12919 external_mini_cluster.cc:1428] Setting key fa26771054e78d4e311ecea6af962678
I20260810 07:25:18.712118 18834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d657f1f0a465440a93702ced1b22f292: Bootstrap starting.
Aug 10 07:25:18 dist-test-slave-ffrd krb5kdc[18704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:18.715083 18834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d657f1f0a465440a93702ced1b22f292: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:18.715977 18834 log.cc:824] T 00000000000000000000000000000000 P d657f1f0a465440a93702ced1b22f292: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:18.718253 18834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d657f1f0a465440a93702ced1b22f292: No bootstrap required, opened a new log
I20260810 07:25:18.720497 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:18.706900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59324 (local address 127.12.157.254:46121)
0810 07:25:18.707312 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:18.707329 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:18.707371 (+ 42us) server_negotiation.cc:407] Connection header received
0810 07:25:18.708211 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:18.708229 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:18.708563 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:18.708843 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:18.709853 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:18.710869 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:18.712039 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:18.712567 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:18.712698 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:18.715606 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:18.715612 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:18.715628 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:18.715662 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:18.717573 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:18.718094 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:18.718099 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:18.718104 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:18.718226 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:18.718532 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:18.718535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:18.718536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:18.718957 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:18.719105 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:18.719431 (+ 326us) server_negotiation.cc:299] Negotiation successful
0810 07:25:18.719762 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":131,"threads_started":1}
I20260810 07:25:18.722569 18834 raft_consensus.cc:348] T 00000000000000000000000000000000 P d657f1f0a465440a93702ced1b22f292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d657f1f0a465440a93702ced1b22f292" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46121 } }
I20260810 07:25:18.722844 18834 raft_consensus.cc:374] T 00000000000000000000000000000000 P d657f1f0a465440a93702ced1b22f292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:18.722940 18834 raft_consensus.cc:729] T 00000000000000000000000000000000 P d657f1f0a465440a93702ced1b22f292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d657f1f0a465440a93702ced1b22f292, State: Initialized, Role: FOLLOWER
I20260810 07:25:18.723397 18834 consensus_queue.cc:260] T 00000000000000000000000000000000 P d657f1f0a465440a93702ced1b22f292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d657f1f0a465440a93702ced1b22f292" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46121 } }
I20260810 07:25:18.723527 18834 raft_consensus.cc:388] T 00000000000000000000000000000000 P d657f1f0a465440a93702ced1b22f292 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:25:18.723608 18834 raft_consensus.cc:482] T 00000000000000000000000000000000 P d657f1f0a465440a93702ced1b22f292 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:25:18.723707 18834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d657f1f0a465440a93702ced1b22f292 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:18.724726 18834 raft_consensus.cc:504] T 00000000000000000000000000000000 P d657f1f0a465440a93702ced1b22f292 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d657f1f0a465440a93702ced1b22f292" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46121 } }
I20260810 07:25:18.725112 18834 leader_election.cc:302] T 00000000000000000000000000000000 P d657f1f0a465440a93702ced1b22f292 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d657f1f0a465440a93702ced1b22f292; no voters:
I20260810 07:25:18.725437 18834 leader_election.cc:288] T 00000000000000000000000000000000 P d657f1f0a465440a93702ced1b22f292 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:25:18.725625 18844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d657f1f0a465440a93702ced1b22f292 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:18.725936 18844 raft_consensus.cc:686] T 00000000000000000000000000000000 P d657f1f0a465440a93702ced1b22f292 [term 1 LEADER]: Becoming Leader. State: Replica: d657f1f0a465440a93702ced1b22f292, State: Running, Role: LEADER
I20260810 07:25:18.726236 18844 consensus_queue.cc:237] T 00000000000000000000000000000000 P d657f1f0a465440a93702ced1b22f292 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d657f1f0a465440a93702ced1b22f292" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46121 } }
I20260810 07:25:18.726713 18834 sys_catalog.cc:564] T 00000000000000000000000000000000 P d657f1f0a465440a93702ced1b22f292 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:25:18.728678 18847 sys_catalog.cc:455] T 00000000000000000000000000000000 P d657f1f0a465440a93702ced1b22f292 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d657f1f0a465440a93702ced1b22f292. Latest consensus state: current_term: 1 leader_uuid: "d657f1f0a465440a93702ced1b22f292" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d657f1f0a465440a93702ced1b22f292" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46121 } } }
I20260810 07:25:18.728824 18847 sys_catalog.cc:458] T 00000000000000000000000000000000 P d657f1f0a465440a93702ced1b22f292 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:18.728696 18845 sys_catalog.cc:455] T 00000000000000000000000000000000 P d657f1f0a465440a93702ced1b22f292 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d657f1f0a465440a93702ced1b22f292" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d657f1f0a465440a93702ced1b22f292" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46121 } } }
I20260810 07:25:18.729378 18845 sys_catalog.cc:458] T 00000000000000000000000000000000 P d657f1f0a465440a93702ced1b22f292 [sys.catalog]: This master's current role is: LEADER
W20260810 07:25:18.731736 18868 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d657f1f0a465440a93702ced1b22f292: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:25:18.731812 18868 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:25:18.731890 18871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:25:18.732563 18871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:25:18.738674 18871 catalog_manager.cc:1347] Generated new cluster ID: 469f9e96f3664cc99455b025f2d925f7
I20260810 07:25:18.738747 18871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:25:18.744223 18871 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:25:18.745213 18871 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:25:18.755056 18871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d657f1f0a465440a93702ced1b22f292: Generated new TSK 0
I20260810 07:25:18.755849 18871 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
I20260810 07:25:18.797569 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:0
--local_ip_for_outbound_sockets=127.12.157.193
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:46121
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:39679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:18.918550 18880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:18.918817 18880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:18.918869 18880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:18.924463 18880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:18.924526 18880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:18.924615 18880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:25:18.930240 18880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:39679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:46121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:18.931378 18880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:18.932428 18880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 07:25:18.942569 18880 server_base.cc:1034] running on GCE node
W20260810 07:25:18.942539 18889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:18.942524 18893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:18.942729 18891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:18.943163 18880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:18.943825 18880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:18.945021 18880 hybrid_clock.cc:648] HybridClock initialized: now 1786346718945000 us; error 31 us; skew 500 ppm
Aug 10 07:25:18 dist-test-slave-ffrd krb5kdc[18704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346718, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:18.947780 18880 init.cc:377] Logged in from keytab as kudu/127.12.157.193@KRBTEST.COM (short username kudu)
I20260810 07:25:18.948697 18880 webserver.cc:460] Webserver started at http://127.12.157.193:39305/ using document root <none> and password file <none>
I20260810 07:25:18.949314 18880 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:18.949416 18880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:18.949646 18880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:18.951656 18880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-0/data/instance:
uuid: "7bda345113dd4329a178cc999af8db88"
format_stamp: "Formatted at 2026-08-10 07:25:18 on dist-test-slave-ffrd"
server_key: "17533da9c1b508d3524c6cf31fe67666"
server_key_iv: "4e2e99b1acf6910257fdfe6020fb6003"
server_key_version: "encryptionkey@0"
I20260810 07:25:18.952215 18880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance:
uuid: "7bda345113dd4329a178cc999af8db88"
format_stamp: "Formatted at 2026-08-10 07:25:18 on dist-test-slave-ffrd"
server_key: "17533da9c1b508d3524c6cf31fe67666"
server_key_iv: "4e2e99b1acf6910257fdfe6020fb6003"
server_key_version: "encryptionkey@0"
I20260810 07:25:18.955896 18880 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 07:25:18.958390 18908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:18.959646 18880 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 07:25:18.959793 18880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "7bda345113dd4329a178cc999af8db88"
format_stamp: "Formatted at 2026-08-10 07:25:18 on dist-test-slave-ffrd"
server_key: "17533da9c1b508d3524c6cf31fe67666"
server_key_iv: "4e2e99b1acf6910257fdfe6020fb6003"
server_key_version: "encryptionkey@0"
I20260810 07:25:18.959929 18880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:18.980044 18880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:18.983574 18880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:18.983805 18880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:18.984529 18880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:18.985528 18880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:18.985608 18880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:18.985687 18880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:18.985734 18880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:18.995404 18880 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:40923
I20260810 07:25:18.995427 19090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:40923 every 8 connection(s)
I20260810 07:25:18.996371 18880 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:25:19.005152 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 18880
I20260810 07:25:19.005317 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance
I20260810 07:25:19.005620 12919 external_mini_cluster.cc:1428] Setting key 3d791783eb9f22f9786646d935cc5c4c
Aug 10 07:25:19 dist-test-slave-ffrd krb5kdc[18704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346718, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:19.011858 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:18.998322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:43005 (local address 127.12.157.254:46121)
0810 07:25:18.998481 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:18.998485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:18.999205 (+ 720us) server_negotiation.cc:407] Connection header received
0810 07:25:19.000678 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:19.000682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:19.000740 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:19.000851 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:19.002080 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.002626 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.003781 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.004032 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.004138 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:19.007387 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:19.007392 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:19.007394 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:19.007440 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:19.009513 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:19.010152 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:19.010156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:19.010158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:19.010252 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:19.010643 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:19.010648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:19.010649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:19.010865 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:19.010978 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:19.011520 (+ 542us) server_negotiation.cc:299] Negotiation successful
0810 07:25:19.011669 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 07:25:19.012825 19098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:18.998585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46121 (local address 127.12.157.193:43005)
0810 07:25:18.999044 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:18.999088 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:19.000465 (+ 1377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:19.001049 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:19.001057 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:19.001425 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:19.001921 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:19.001935 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.002752 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.002755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:19.003650 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:19.003659 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.004201 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.004205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:19.004384 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:19.005140 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:19.005160 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:19.007083 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:19.009659 (+ 2576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:19.009665 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:19.009681 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:19.010031 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:19.010388 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:19.010391 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:19.010392 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:19.010542 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:19.010978 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:19.010987 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:19.011267 (+ 280us) client_negotiation.cc:769] Sending connection context
0810 07:25:19.011479 (+ 212us) client_negotiation.cc:240] Negotiation successful
0810 07:25:19.011849 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":112,"threads_started":1}
I20260810 07:25:19.014217 19095 heartbeater.cc:347] Connected to a master server at 127.12.157.254:46121
I20260810 07:25:19.014561 19095 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:19.015232 19095 heartbeater.cc:511] Master 127.12.157.254:46121 requested a full tablet report, sending...
I20260810 07:25:19.017206 18783 ts_manager.cc:194] Registered new tserver with Master: 7bda345113dd4329a178cc999af8db88 (127.12.157.193:40923)
WARNING: no policy specified for kudu/127.12.157.194@KRBTEST.COM; defaulting to no policy
I20260810 07:25:19.018733 18783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.193@KRBTEST.COM'} at 127.12.157.193:43005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
I20260810 07:25:19.040831 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:0
--local_ip_for_outbound_sockets=127.12.157.194
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:46121
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:39679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:19.163178 19101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:19.163466 19101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:19.163513 19101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:19.167646 19101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:19.167717 19101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:19.167785 19101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:25:19.173393 19101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:39679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:46121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:19.174474 19101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:19.175480 19101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 07:25:19.185590 19101 server_base.cc:1034] running on GCE node
W20260810 07:25:19.185489 19108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:19.185506 19110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:19.185487 19112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:19.186184 19101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:19.186887 19101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:19.188083 19101 hybrid_clock.cc:648] HybridClock initialized: now 1786346719188063 us; error 29 us; skew 500 ppm
Aug 10 07:25:19 dist-test-slave-ffrd krb5kdc[18704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346719, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:19.191143 19101 init.cc:377] Logged in from keytab as kudu/127.12.157.194@KRBTEST.COM (short username kudu)
I20260810 07:25:19.192039 19101 webserver.cc:460] Webserver started at http://127.12.157.194:40427/ using document root <none> and password file <none>
I20260810 07:25:19.192619 19101 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:19.192713 19101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:19.193020 19101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:19.195103 19101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-1/data/instance:
uuid: "6c9266b02eba41f7ab0080bc0d225e1f"
format_stamp: "Formatted at 2026-08-10 07:25:19 on dist-test-slave-ffrd"
server_key: "4ea293cc72c4e5805e8a0996dd445059"
server_key_iv: "fa766970bdafae1ea76050ef09f94d16"
server_key_version: "encryptionkey@0"
I20260810 07:25:19.195680 19101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance:
uuid: "6c9266b02eba41f7ab0080bc0d225e1f"
format_stamp: "Formatted at 2026-08-10 07:25:19 on dist-test-slave-ffrd"
server_key: "4ea293cc72c4e5805e8a0996dd445059"
server_key_iv: "fa766970bdafae1ea76050ef09f94d16"
server_key_version: "encryptionkey@0"
I20260810 07:25:19.199585 19101 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260810 07:25:19.202186 19129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:19.203449 19101 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 07:25:19.203596 19101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "6c9266b02eba41f7ab0080bc0d225e1f"
format_stamp: "Formatted at 2026-08-10 07:25:19 on dist-test-slave-ffrd"
server_key: "4ea293cc72c4e5805e8a0996dd445059"
server_key_iv: "fa766970bdafae1ea76050ef09f94d16"
server_key_version: "encryptionkey@0"
I20260810 07:25:19.203718 19101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:19.229763 19101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:19.233141 19101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:19.233367 19101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:19.233995 19101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:19.234874 19101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:19.234942 19101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:19.235010 19101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:19.235057 19101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:19.244675 19101 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:36993
I20260810 07:25:19.244714 19312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:36993 every 8 connection(s)
I20260810 07:25:19.245671 19101 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:25:19.249601 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 19101
I20260810 07:25:19.249747 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance
I20260810 07:25:19.250025 12919 external_mini_cluster.cc:1428] Setting key 6488b9e658eecfaa74a023bcf76e7a73
Aug 10 07:25:19 dist-test-slave-ffrd krb5kdc[18704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346719, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.157.195@KRBTEST.COM; defaulting to no policy
I20260810 07:25:19.260398 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:19.247747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:34271 (local address 127.12.157.254:46121)
0810 07:25:19.247916 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:19.247921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:19.248804 (+ 883us) server_negotiation.cc:407] Connection header received
0810 07:25:19.250426 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:19.250430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:19.250488 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:19.250576 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:19.251780 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.252304 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.253645 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.253871 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.253981 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:19.256662 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:19.256666 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:19.256668 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:19.256692 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:19.258192 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:19.258736 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:19.258740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:19.258741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:19.258827 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:19.259182 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:19.259187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:19.259188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:19.259396 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:19.259507 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:19.260065 (+ 558us) server_negotiation.cc:299] Negotiation successful
0810 07:25:19.260203 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 07:25:19.261387 19316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:19.248136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46121 (local address 127.12.157.194:34271)
0810 07:25:19.248644 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:19.248688 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:19.250220 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:19.250772 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:19.250780 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:19.251151 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:19.251638 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:19.251652 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.252445 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.252449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:19.253515 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:19.253529 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.254026 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.254030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:19.254208 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:19.254907 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:19.254925 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:19.256472 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:19.258321 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:19.258328 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:19.258345 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:19.258624 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:19.258923 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:19.258926 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:19.258928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:19.259085 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:19.259507 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:19.259518 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:19.259793 (+ 275us) client_negotiation.cc:769] Sending connection context
0810 07:25:19.260006 (+ 213us) client_negotiation.cc:240] Negotiation successful
0810 07:25:19.260375 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":130,"threads_started":1}
I20260810 07:25:19.262727 19313 heartbeater.cc:347] Connected to a master server at 127.12.157.254:46121
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.195@KRBTEST.COM" created.
I20260810 07:25:19.263041 19313 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:19.263640 19313 heartbeater.cc:511] Master 127.12.157.254:46121 requested a full tablet report, sending...
I20260810 07:25:19.264904 18783 ts_manager.cc:194] Registered new tserver with Master: 6c9266b02eba41f7ab0080bc0d225e1f (127.12.157.194:36993)
I20260810 07:25:19.265630 18783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.194@KRBTEST.COM'} at 127.12.157.194:34271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
I20260810 07:25:19.280059 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:0
--local_ip_for_outbound_sockets=127.12.157.195
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:46121
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:39679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:19.401237 19321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:19.401515 19321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:19.401561 19321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:19.405530 19321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:19.405593 19321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:19.405652 19321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:25:19.411810 19321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:39679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:46121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:19.413138 19321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:19.414250 19321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:19.424815 19329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:19.424868 19328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:19.424827 19331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:19.425153 19321 server_base.cc:1034] running on GCE node
I20260810 07:25:19.425803 19321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:19.426560 19321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:19.427749 19321 hybrid_clock.cc:648] HybridClock initialized: now 1786346719427719 us; error 43 us; skew 500 ppm
Aug 10 07:25:19 dist-test-slave-ffrd krb5kdc[18704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346719, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:19.430706 19321 init.cc:377] Logged in from keytab as kudu/127.12.157.195@KRBTEST.COM (short username kudu)
I20260810 07:25:19.431578 19321 webserver.cc:460] Webserver started at http://127.12.157.195:37001/ using document root <none> and password file <none>
I20260810 07:25:19.432114 19321 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:19.432173 19321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:19.432343 19321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:19.434630 19321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-2/data/instance:
uuid: "1bf5d268b6004de4814d4e88b8098ad9"
format_stamp: "Formatted at 2026-08-10 07:25:19 on dist-test-slave-ffrd"
server_key: "1d49c93c16b42efab50be4b4a2130ac0"
server_key_iv: "67026048a4e87d2b4e685b8ce8a024df"
server_key_version: "encryptionkey@0"
I20260810 07:25:19.435187 19321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance:
uuid: "1bf5d268b6004de4814d4e88b8098ad9"
format_stamp: "Formatted at 2026-08-10 07:25:19 on dist-test-slave-ffrd"
server_key: "1d49c93c16b42efab50be4b4a2130ac0"
server_key_iv: "67026048a4e87d2b4e685b8ce8a024df"
server_key_version: "encryptionkey@0"
I20260810 07:25:19.439041 19321 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 07:25:19.441506 19348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:19.442780 19321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:25:19.442936 19321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "1bf5d268b6004de4814d4e88b8098ad9"
format_stamp: "Formatted at 2026-08-10 07:25:19 on dist-test-slave-ffrd"
server_key: "1d49c93c16b42efab50be4b4a2130ac0"
server_key_iv: "67026048a4e87d2b4e685b8ce8a024df"
server_key_version: "encryptionkey@0"
I20260810 07:25:19.443056 19321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:19.465004 19321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:19.467976 19321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:19.468279 19321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:19.469000 19321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:19.469858 19321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:19.469915 19321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:19.469956 19321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:19.470007 19321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:19.479149 19321 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:42583
I20260810 07:25:19.479171 19522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:42583 every 8 connection(s)
I20260810 07:25:19.480152 19321 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:25:19.488746 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 19321
I20260810 07:25:19.488914 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance
I20260810 07:25:19.489275 12919 external_mini_cluster.cc:1428] Setting key 3763e3163c9e04d09f21ce9e883920ea
Aug 10 07:25:19 dist-test-slave-ffrd krb5kdc[18704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346719, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:19.498647 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:19.482095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:51361 (local address 127.12.157.254:46121)
0810 07:25:19.482243 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:19.482248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:19.485537 (+ 3289us) server_negotiation.cc:407] Connection header received
0810 07:25:19.487385 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:19.487395 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:19.487476 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:19.487570 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:19.489362 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.490211 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.491965 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.492191 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.492274 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:19.494875 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:19.494878 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:19.494880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:19.494907 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:19.496438 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:19.497027 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:19.497030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:19.497031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:19.497114 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:19.497523 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:19.497527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:19.497528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:19.497761 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:19.497853 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:19.498408 (+ 555us) server_negotiation.cc:299] Negotiation successful
0810 07:25:19.498520 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 07:25:19.499543 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:19.482634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46121 (local address 127.12.157.195:51361)
0810 07:25:19.485321 (+ 2687us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:19.485404 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:19.487059 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:19.487859 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:19.487870 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:19.488366 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:19.489163 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:19.489186 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.490388 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.490391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:19.491820 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:19.491831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.492335 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.492337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:19.492497 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:19.493188 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:19.493211 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:19.494706 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:19.496580 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:19.496586 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:19.496600 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:19.496869 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:19.497203 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:19.497206 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:19.497207 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:19.497352 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:19.497864 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:19.497870 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:19.498138 (+ 268us) client_negotiation.cc:769] Sending connection context
0810 07:25:19.498296 (+ 158us) client_negotiation.cc:240] Negotiation successful
0810 07:25:19.498575 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2416,"thread_start_us":182,"threads_started":1}
I20260810 07:25:19.501060 19523 heartbeater.cc:347] Connected to a master server at 127.12.157.254:46121
I20260810 07:25:19.501376 19523 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:19.502002 19523 heartbeater.cc:511] Master 127.12.157.254:46121 requested a full tablet report, sending...
I20260810 07:25:19.503249 18783 ts_manager.cc:194] Registered new tserver with Master: 1bf5d268b6004de4814d4e88b8098ad9 (127.12.157.195:42583)
I20260810 07:25:19.503881 18783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.195@KRBTEST.COM'} at 127.12.157.195:51361
I20260810 07:25:19.504056 12919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 07:25:19 dist-test-slave-ffrd krb5kdc[18704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 07:25:19 dist-test-slave-ffrd krb5kdc[18704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.193@KRBTEST.COM
I20260810 07:25:19.535653 19532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:19.524547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54422 (local address 127.12.157.193:40923)
0810 07:25:19.524895 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:19.524903 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:19.524928 (+ 25us) server_negotiation.cc:407] Connection header received
0810 07:25:19.525043 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:19.525049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:19.525260 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:19.525419 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:19.526090 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.526987 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.528024 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.528527 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.528609 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:19.531688 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:19.531700 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:19.531714 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:19.531742 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:19.533605 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:19.534102 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:19.534107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:19.534109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:19.534192 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:19.534561 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:19.534563 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:19.534564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:19.534814 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:19.534980 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:19.535187 (+ 207us) server_negotiation.cc:299] Negotiation successful
0810 07:25:19.535428 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":116,"threads_started":1}
Aug 10 07:25:19 dist-test-slave-ffrd krb5kdc[18704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:19.547889 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:19.538832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59326 (local address 127.12.157.254:46121)
0810 07:25:19.538994 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:19.539002 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:19.539053 (+ 51us) server_negotiation.cc:407] Connection header received
0810 07:25:19.539197 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:19.539202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:19.539262 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:19.539354 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:19.539983 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.540496 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.541405 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:19.541634 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:19.541724 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:19.544231 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:19.544234 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:19.544236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:19.544265 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:19.545873 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:19.546347 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:19.546351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:19.546353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:19.546451 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:19.546768 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:19.546771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:19.546772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:19.547082 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:19.547196 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:19.547514 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 07:25:19.547685 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260810 07:25:19.548373 18783 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59326
I20260810 07:25:19.549141 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 18880
I20260810 07:25:19.564447 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 19101
I20260810 07:25:19.574122 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 19321
I20260810 07:25:19.589251 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 18727
2026-08-10T07:25:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2504 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:25:19 dist-test-slave-ffrd krb5kdc[19539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:25:19 dist-test-slave-ffrd krb5kdc[19539](info): set up 2 sockets
Aug 10 07:25:19 dist-test-slave-ffrd krb5kdc[19539](info): commencing operation
krb5kdc: starting...
W20260810 07:25:20.858618 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.240s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:25:20 dist-test-slave-ffrd krb5kdc[19539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:25:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:25:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:25:20.989203 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:37371
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:42553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:37371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:21.105324 19557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:21.105650 19557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:21.105726 19557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:21.109851 19557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:25:21.109961 19557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:21.109997 19557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:21.110035 19557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:25:21.110066 19557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:25:21.115613 19557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:42553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:37371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:37371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:21.116813 19557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:21.117939 19557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:21.126369 19568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:21.126374 19565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:21.126363 19570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:21.126652 19557 server_base.cc:1034] running on GCE node
I20260810 07:25:21.127290 19557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:21.128510 19557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:21.129721 19557 hybrid_clock.cc:648] HybridClock initialized: now 1786346721129692 us; error 45 us; skew 500 ppm
Aug 10 07:25:21 dist-test-slave-ffrd krb5kdc[19539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346721, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:21.132616 19557 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:25:21.133451 19557 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:21.133519 19557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:21.133695 19557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:21.135903 19557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "4086d063886b48c6923c1df7c9ae1df8"
format_stamp: "Formatted at 2026-08-10 07:25:21 on dist-test-slave-ffrd"
server_key: "85fa063bad9d05ae2a91c4b73fc19777"
server_key_iv: "a9da91df6b7aedd86142d67d1649cd31"
server_key_version: "encryptionkey@0"
I20260810 07:25:21.136477 19557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "4086d063886b48c6923c1df7c9ae1df8"
format_stamp: "Formatted at 2026-08-10 07:25:21 on dist-test-slave-ffrd"
server_key: "85fa063bad9d05ae2a91c4b73fc19777"
server_key_iv: "a9da91df6b7aedd86142d67d1649cd31"
server_key_version: "encryptionkey@0"
I20260810 07:25:21.140851 19557 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 07:25:21.143527 19577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:21.144883 19557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 07:25:21.145079 19557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "4086d063886b48c6923c1df7c9ae1df8"
format_stamp: "Formatted at 2026-08-10 07:25:21 on dist-test-slave-ffrd"
server_key: "85fa063bad9d05ae2a91c4b73fc19777"
server_key_iv: "a9da91df6b7aedd86142d67d1649cd31"
server_key_version: "encryptionkey@0"
I20260810 07:25:21.145195 19557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:21.168066 19557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:21.171149 19557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:21.171375 19557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:21.178738 19557 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:37371
I20260810 07:25:21.179049 19667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:37371 every 8 connection(s)
I20260810 07:25:21.179441 19557 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:25:21.182513 19669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:21.186163 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 19557
I20260810 07:25:21.186312 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:25:21.186620 12919 external_mini_cluster.cc:1428] Setting key afd02c1187b72f8400bbee9d15ebbd5d
I20260810 07:25:21.189297 19669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4086d063886b48c6923c1df7c9ae1df8: Bootstrap starting.
I20260810 07:25:21.191534 19669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4086d063886b48c6923c1df7c9ae1df8: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:21.192330 19669 log.cc:824] T 00000000000000000000000000000000 P 4086d063886b48c6923c1df7c9ae1df8: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:21.194654 19669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4086d063886b48c6923c1df7c9ae1df8: No bootstrap required, opened a new log
Aug 10 07:25:21 dist-test-slave-ffrd krb5kdc[19539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:21.198593 19669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4086d063886b48c6923c1df7c9ae1df8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4086d063886b48c6923c1df7c9ae1df8" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37371 } }
I20260810 07:25:21.198925 19669 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4086d063886b48c6923c1df7c9ae1df8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:21.198988 19669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4086d063886b48c6923c1df7c9ae1df8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4086d063886b48c6923c1df7c9ae1df8, State: Initialized, Role: FOLLOWER
I20260810 07:25:21.199515 19669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4086d063886b48c6923c1df7c9ae1df8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4086d063886b48c6923c1df7c9ae1df8" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37371 } }
I20260810 07:25:21.199668 19669 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4086d063886b48c6923c1df7c9ae1df8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:25:21.199748 19669 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4086d063886b48c6923c1df7c9ae1df8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:25:21.199878 19669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4086d063886b48c6923c1df7c9ae1df8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:21.201159 19669 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4086d063886b48c6923c1df7c9ae1df8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4086d063886b48c6923c1df7c9ae1df8" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37371 } }
I20260810 07:25:21.201512 19669 leader_election.cc:302] T 00000000000000000000000000000000 P 4086d063886b48c6923c1df7c9ae1df8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4086d063886b48c6923c1df7c9ae1df8; no voters:
I20260810 07:25:21.201838 19669 leader_election.cc:288] T 00000000000000000000000000000000 P 4086d063886b48c6923c1df7c9ae1df8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:25:21.201960 19677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4086d063886b48c6923c1df7c9ae1df8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:21.202062 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:21.188120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38280 (local address 127.12.157.254:37371)
0810 07:25:21.188619 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:21.188635 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:21.188672 (+ 37us) server_negotiation.cc:407] Connection header received
0810 07:25:21.189454 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:21.189474 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:21.189804 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:21.190145 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:21.190991 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:21.192075 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:21.193084 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:21.193563 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:21.193737 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:21.196527 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:21.196533 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:21.196547 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:21.196574 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:21.198579 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:21.199116 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:21.199122 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:21.199129 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:21.199256 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:21.199557 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:21.199561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:21.199562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:21.200148 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:21.200327 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:21.200661 (+ 334us) server_negotiation.cc:299] Negotiation successful
0810 07:25:21.201122 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":166,"threads_started":1}
I20260810 07:25:21.202183 19677 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4086d063886b48c6923c1df7c9ae1df8 [term 1 LEADER]: Becoming Leader. State: Replica: 4086d063886b48c6923c1df7c9ae1df8, State: Running, Role: LEADER
I20260810 07:25:21.202533 19677 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4086d063886b48c6923c1df7c9ae1df8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4086d063886b48c6923c1df7c9ae1df8" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37371 } }
I20260810 07:25:21.203061 19669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4086d063886b48c6923c1df7c9ae1df8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:25:21.203701 19679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4086d063886b48c6923c1df7c9ae1df8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4086d063886b48c6923c1df7c9ae1df8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4086d063886b48c6923c1df7c9ae1df8" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37371 } } }
I20260810 07:25:21.203801 19679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4086d063886b48c6923c1df7c9ae1df8 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:21.204080 19681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4086d063886b48c6923c1df7c9ae1df8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4086d063886b48c6923c1df7c9ae1df8. Latest consensus state: current_term: 1 leader_uuid: "4086d063886b48c6923c1df7c9ae1df8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4086d063886b48c6923c1df7c9ae1df8" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37371 } } }
I20260810 07:25:21.204164 19681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4086d063886b48c6923c1df7c9ae1df8 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:21.204607 19687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:25:21.207804 19687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:25:21.213954 19687 catalog_manager.cc:1347] Generated new cluster ID: 6dd7a26c19c449b0ab46ac23f8a2134c
I20260810 07:25:21.214049 19687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:25:21.222575 19687 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:25:21.224119 19687 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:25:21.240164 19687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4086d063886b48c6923c1df7c9ae1df8: Generated new TSK 0
I20260810 07:25:21.240994 19687 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
I20260810 07:25:21.280025 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:0
--local_ip_for_outbound_sockets=127.12.157.193
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37371
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:42553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:21.409657 19709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:21.409948 19709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:21.410005 19709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:21.414255 19709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:21.414330 19709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:21.414394 19709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:25:21.420161 19709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:42553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:21.421248 19709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:21.422238 19709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:21.432364 19716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:21.432384 19715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:21.432384 19719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:21.432611 19709 server_base.cc:1034] running on GCE node
I20260810 07:25:21.433276 19709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:21.434020 19709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:21.435194 19709 hybrid_clock.cc:648] HybridClock initialized: now 1786346721435162 us; error 46 us; skew 500 ppm
Aug 10 07:25:21 dist-test-slave-ffrd krb5kdc[19539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346721, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:21.438261 19709 init.cc:377] Logged in from keytab as kudu/127.12.157.193@KRBTEST.COM (short username kudu)
I20260810 07:25:21.438968 19709 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:21.439033 19709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:21.439208 19709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:21.441380 19709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-0/data/instance:
uuid: "e8a9b6d1d5334560b1ccf07ff07ea9a0"
format_stamp: "Formatted at 2026-08-10 07:25:21 on dist-test-slave-ffrd"
server_key: "593ab6c3bdde9d97d49392a7ddbe9b37"
server_key_iv: "b5b627daed52df6d4bdb997f742a03a5"
server_key_version: "encryptionkey@0"
I20260810 07:25:21.441896 19709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance:
uuid: "e8a9b6d1d5334560b1ccf07ff07ea9a0"
format_stamp: "Formatted at 2026-08-10 07:25:21 on dist-test-slave-ffrd"
server_key: "593ab6c3bdde9d97d49392a7ddbe9b37"
server_key_iv: "b5b627daed52df6d4bdb997f742a03a5"
server_key_version: "encryptionkey@0"
I20260810 07:25:21.445693 19709 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 07:25:21.448235 19731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:21.449720 19709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:25:21.449841 19709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "e8a9b6d1d5334560b1ccf07ff07ea9a0"
format_stamp: "Formatted at 2026-08-10 07:25:21 on dist-test-slave-ffrd"
server_key: "593ab6c3bdde9d97d49392a7ddbe9b37"
server_key_iv: "b5b627daed52df6d4bdb997f742a03a5"
server_key_version: "encryptionkey@0"
I20260810 07:25:21.449947 19709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:21.473090 19709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:21.477209 19709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:21.477478 19709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:21.478196 19709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:21.479178 19709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:21.479231 19709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:21.479274 19709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:21.479290 19709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:21.490854 19709 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:40943
I20260810 07:25:21.490872 19920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:40943 every 8 connection(s)
I20260810 07:25:21.491641 19709 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:25:21.499238 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 19709
I20260810 07:25:21.499437 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance
I20260810 07:25:21.499737 12919 external_mini_cluster.cc:1428] Setting key 73109ce997f4b7bdfeb9b88df794b11d
Aug 10 07:25:21 dist-test-slave-ffrd krb5kdc[19539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346721, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:21.507375 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:21.494148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:33003 (local address 127.12.157.254:37371)
0810 07:25:21.494325 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:21.494329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:21.495208 (+ 879us) server_negotiation.cc:407] Connection header received
0810 07:25:21.496527 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:21.496530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:21.496592 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:21.496695 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:21.497967 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:21.498517 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:21.499785 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:21.500053 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:21.500165 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:21.503098 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:21.503105 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:21.503111 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:21.503147 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:21.504871 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:21.505712 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:21.505719 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:21.505722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:21.505813 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:21.506179 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:21.506183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:21.506184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:21.506380 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:21.506516 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:21.507065 (+ 549us) server_negotiation.cc:299] Negotiation successful
0810 07:25:21.507200 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 07:25:21.508270 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:21.494519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37371 (local address 127.12.157.193:33003)
0810 07:25:21.495040 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:21.495087 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:21.496277 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:21.496876 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:21.496882 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:21.497304 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:21.497813 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:21.497828 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:21.498647 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:21.498650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:21.499664 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:21.499675 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:21.500215 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:21.500220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:21.500393 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:21.501083 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:21.501101 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:21.502887 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:21.505150 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:21.505160 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:21.505173 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:21.505569 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:21.505911 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:21.505914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:21.505916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:21.506072 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:21.506523 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:21.506533 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:21.506796 (+ 263us) client_negotiation.cc:769] Sending connection context
0810 07:25:21.507008 (+ 212us) client_negotiation.cc:240] Negotiation successful
0810 07:25:21.507359 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":127,"threads_started":1}
I20260810 07:25:21.510095 19922 heartbeater.cc:347] Connected to a master server at 127.12.157.254:37371
I20260810 07:25:21.510519 19922 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:21.511317 19922 heartbeater.cc:511] Master 127.12.157.254:37371 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.157.194@KRBTEST.COM; defaulting to no policy
I20260810 07:25:21.513530 19602 ts_manager.cc:194] Registered new tserver with Master: e8a9b6d1d5334560b1ccf07ff07ea9a0 (127.12.157.193:40943)
I20260810 07:25:21.514825 19602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.193@KRBTEST.COM'} at 127.12.157.193:33003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
I20260810 07:25:21.532230 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:0
--local_ip_for_outbound_sockets=127.12.157.194
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37371
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:42553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:21.648432 19931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:21.648732 19931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:21.648840 19931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:21.653151 19931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:21.653244 19931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:21.653342 19931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:25:21.658887 19931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:42553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:21.660058 19931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:21.661207 19931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:21.671422 19940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:21.671433 19942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:21.671617 19931 server_base.cc:1034] running on GCE node
W20260810 07:25:21.671422 19939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:21.672132 19931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:21.672858 19931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:21.674029 19931 hybrid_clock.cc:648] HybridClock initialized: now 1786346721674006 us; error 35 us; skew 500 ppm
Aug 10 07:25:21 dist-test-slave-ffrd krb5kdc[19539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346721, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:21.676899 19931 init.cc:377] Logged in from keytab as kudu/127.12.157.194@KRBTEST.COM (short username kudu)
I20260810 07:25:21.677717 19931 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:21.677815 19931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:21.678052 19931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:21.680228 19931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-1/data/instance:
uuid: "83b66c627600491c92600d4ec1593740"
format_stamp: "Formatted at 2026-08-10 07:25:21 on dist-test-slave-ffrd"
server_key: "c3e8fb3c2dad64b2ab503501ab39605b"
server_key_iv: "8dfeb9208a9c25f25e489961c0c99747"
server_key_version: "encryptionkey@0"
I20260810 07:25:21.680821 19931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance:
uuid: "83b66c627600491c92600d4ec1593740"
format_stamp: "Formatted at 2026-08-10 07:25:21 on dist-test-slave-ffrd"
server_key: "c3e8fb3c2dad64b2ab503501ab39605b"
server_key_iv: "8dfeb9208a9c25f25e489961c0c99747"
server_key_version: "encryptionkey@0"
I20260810 07:25:21.684607 19931 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 07:25:21.687167 19961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:21.688421 19931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 07:25:21.688575 19931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "83b66c627600491c92600d4ec1593740"
format_stamp: "Formatted at 2026-08-10 07:25:21 on dist-test-slave-ffrd"
server_key: "c3e8fb3c2dad64b2ab503501ab39605b"
server_key_iv: "8dfeb9208a9c25f25e489961c0c99747"
server_key_version: "encryptionkey@0"
I20260810 07:25:21.688686 19931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:21.705034 19931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:21.708348 19931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:21.708556 19931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:21.709249 19931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:21.710177 19931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:21.710246 19931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:21.710319 19931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:21.710366 19931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:21.719934 19931 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:37183
I20260810 07:25:21.719954 20155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:37183 every 8 connection(s)
I20260810 07:25:21.720620 19931 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
Aug 10 07:25:21 dist-test-slave-ffrd krb5kdc[19539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346721, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:21.730425 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 19931
I20260810 07:25:21.730563 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance
I20260810 07:25:21.731025 12919 external_mini_cluster.cc:1428] Setting key e9c2d11607874e98817a1f2b81134a71
I20260810 07:25:21.734901 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:21.722609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:40869 (local address 127.12.157.254:37371)
0810 07:25:21.722796 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:21.722801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:21.723634 (+ 833us) server_negotiation.cc:407] Connection header received
0810 07:25:21.725017 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:21.725020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:21.725072 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:21.725159 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:21.726334 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:21.726902 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:21.727992 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:21.728205 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:21.728283 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:21.730872 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:21.730877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:21.730880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:21.730912 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:21.732613 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:21.733249 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:21.733253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:21.733255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:21.733340 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:21.733703 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:21.733707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:21.733708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:21.733914 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:21.734023 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:21.734579 (+ 556us) server_negotiation.cc:299] Negotiation successful
0810 07:25:21.734722 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 07:25:21.735797 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:21.722976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37371 (local address 127.12.157.194:40869)
0810 07:25:21.723462 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:21.723519 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:21.724748 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:21.725320 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:21.725327 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:21.725691 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:21.726201 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:21.726215 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:21.727015 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:21.727017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:21.727891 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:21.727900 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:21.728334 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:21.728336 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:21.728479 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:21.729133 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:21.729150 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:21.730658 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:21.732749 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:21.732756 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:21.732773 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:21.733145 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:21.733434 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:21.733437 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:21.733439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:21.733605 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:21.734037 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:21.734045 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:21.734322 (+ 277us) client_negotiation.cc:769] Sending connection context
0810 07:25:21.734533 (+ 211us) client_negotiation.cc:240] Negotiation successful
0810 07:25:21.734900 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":132,"threads_started":1}
I20260810 07:25:21.737242 20156 heartbeater.cc:347] Connected to a master server at 127.12.157.254:37371
I20260810 07:25:21.737552 20156 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:21.738070 20156 heartbeater.cc:511] Master 127.12.157.254:37371 requested a full tablet report, sending...
I20260810 07:25:21.739187 19602 ts_manager.cc:194] Registered new tserver with Master: 83b66c627600491c92600d4ec1593740 (127.12.157.194:37183)
I20260810 07:25:21.739794 19602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.194@KRBTEST.COM'} at 127.12.157.194:40869
WARNING: no policy specified for kudu/127.12.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
I20260810 07:25:21.761695 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:0
--local_ip_for_outbound_sockets=127.12.157.195
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37371
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:42553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:21.884914 20161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:21.885244 20161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:21.885294 20161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:21.889482 20161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:21.889549 20161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:21.889652 20161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:25:21.895182 20161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:42553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:21.896314 20161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:21.897399 20161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:21.907264 20172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:21.907258 20173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:21.907302 20177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:21.908540 20161 server_base.cc:1034] running on GCE node
I20260810 07:25:21.909202 20161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:21.909965 20161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:21.911154 20161 hybrid_clock.cc:648] HybridClock initialized: now 1786346721911137 us; error 35 us; skew 500 ppm
Aug 10 07:25:21 dist-test-slave-ffrd krb5kdc[19539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346721, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:21.914155 20161 init.cc:377] Logged in from keytab as kudu/127.12.157.195@KRBTEST.COM (short username kudu)
I20260810 07:25:21.914908 20161 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:21.915001 20161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:21.915220 20161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:21.917418 20161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-2/data/instance:
uuid: "586ed486bb8b49fa9db378dda19d5125"
format_stamp: "Formatted at 2026-08-10 07:25:21 on dist-test-slave-ffrd"
server_key: "44e7d6fe09a4244983c748153c03fc91"
server_key_iv: "a05ca7f9e855e2c86ca88024b86b90af"
server_key_version: "encryptionkey@0"
I20260810 07:25:21.918017 20161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance:
uuid: "586ed486bb8b49fa9db378dda19d5125"
format_stamp: "Formatted at 2026-08-10 07:25:21 on dist-test-slave-ffrd"
server_key: "44e7d6fe09a4244983c748153c03fc91"
server_key_iv: "a05ca7f9e855e2c86ca88024b86b90af"
server_key_version: "encryptionkey@0"
I20260810 07:25:21.921904 20161 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 07:25:21.924531 20187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:21.926213 20161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:25:21.926378 20161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "586ed486bb8b49fa9db378dda19d5125"
format_stamp: "Formatted at 2026-08-10 07:25:21 on dist-test-slave-ffrd"
server_key: "44e7d6fe09a4244983c748153c03fc91"
server_key_iv: "a05ca7f9e855e2c86ca88024b86b90af"
server_key_version: "encryptionkey@0"
I20260810 07:25:21.926504 20161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:21.942391 20161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:21.945703 20161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:21.945930 20161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:21.946564 20161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:21.947441 20161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:21.947510 20161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:21.947602 20161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:21.947651 20161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:21.957291 20161 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:34435
I20260810 07:25:21.957314 20383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:34435 every 8 connection(s)
I20260810 07:25:21.957885 20161 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:25:21.959336 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 20161
I20260810 07:25:21.959484 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance
I20260810 07:25:21.959765 12919 external_mini_cluster.cc:1428] Setting key 6ecdfcd4238e0e63a9ed623f1629d6bb
Aug 10 07:25:21 dist-test-slave-ffrd krb5kdc[19539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346721, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:21.971256 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:21.960128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:44935 (local address 127.12.157.254:37371)
0810 07:25:21.960291 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:21.960296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:21.960806 (+ 510us) server_negotiation.cc:407] Connection header received
0810 07:25:21.962349 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:21.962360 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:21.962408 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:21.962478 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:21.963615 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:21.964141 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:21.965316 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:21.965531 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:21.965613 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:21.967979 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:21.967982 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:21.967984 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:21.968008 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:21.969464 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:21.969928 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:21.969930 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:21.969932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:21.970010 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:21.970290 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:21.970292 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:21.970293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:21.970474 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:21.970552 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:21.971037 (+ 485us) server_negotiation.cc:299] Negotiation successful
0810 07:25:21.971135 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 07:25:21.972149 20387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:21.960115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37371 (local address 127.12.157.195:44935)
0810 07:25:21.960664 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:21.960721 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:21.962153 (+ 1432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:21.962633 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:21.962640 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:21.963009 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:21.963479 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:21.963491 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:21.964249 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:21.964252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:21.965209 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:21.965219 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:21.965657 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:21.965659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:21.965792 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:21.966344 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:21.966361 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:21.967816 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:21.969571 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:21.969576 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:21.969587 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:21.969833 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:21.970084 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:21.970086 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:21.970087 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:21.970213 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:21.970585 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:21.970590 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:21.970812 (+ 222us) client_negotiation.cc:769] Sending connection context
0810 07:25:21.970982 (+ 170us) client_negotiation.cc:240] Negotiation successful
0810 07:25:21.971221 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":137,"threads_started":1}
I20260810 07:25:21.973351 20384 heartbeater.cc:347] Connected to a master server at 127.12.157.254:37371
I20260810 07:25:21.973636 20384 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:21.974102 20384 heartbeater.cc:511] Master 127.12.157.254:37371 requested a full tablet report, sending...
I20260810 07:25:21.975097 19601 ts_manager.cc:194] Registered new tserver with Master: 586ed486bb8b49fa9db378dda19d5125 (127.12.157.195:34435)
I20260810 07:25:21.975626 19601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.195@KRBTEST.COM'} at 127.12.157.195:44935
I20260810 07:25:21.985500 12919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:25:21.995003 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:21.987431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38290 (local address 127.12.157.254:37371)
0810 07:25:21.987579 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:21.987584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:21.987764 (+ 180us) server_negotiation.cc:407] Connection header received
0810 07:25:21.987916 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:21.987920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:21.987974 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:21.988055 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:21.988813 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:21.989397 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:21.990233 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:21.990453 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:21.990563 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:21.991889 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:21.991915 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:21.991920 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:21.991958 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:21.993326 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:21.993742 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:21.993747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:21.993748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:21.993834 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:21.994106 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:21.994110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:21.994112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:21.994332 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:21.994475 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:21.994714 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 07:25:21.994827 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 07:25:21.997771 19601 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38290:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:25:22.000746 19601 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:25:22.015650 20404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.009866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43466 (local address 127.12.157.193:40943)
0810 07:25:22.010325 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:22.010336 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:22.010894 (+ 558us) server_negotiation.cc:407] Connection header received
0810 07:25:22.010985 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:22.010994 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:22.011234 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:22.011404 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:22.011689 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.012601 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.014152 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.014914 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.015048 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:22.015183 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:22.015367 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 07:25:22.015470 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":133,"threads_started":1}
I20260810 07:25:22.015657 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.009668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:40943 (local address 127.0.0.1:43466)
0810 07:25:22.010669 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:22.010683 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:22.010788 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:22.011404 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:22.011411 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:22.011433 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:22.011575 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:22.011582 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.012711 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.012716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:22.014032 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:22.014041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.015045 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.015051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:22.015173 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:22.015214 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 07:25:22.015335 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 07:25:22.015487 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"mutex_wait_us":27,"thread_start_us":103,"threads_started":1}
I20260810 07:25:22.016043 20406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.010588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52696 (local address 127.12.157.194:37183)
0810 07:25:22.011006 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:22.011019 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:22.011046 (+ 27us) server_negotiation.cc:407] Connection header received
0810 07:25:22.011132 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:22.011137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:22.011319 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:22.011587 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:22.012157 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.012931 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.014458 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.015026 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.015166 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:22.015297 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:22.015752 (+ 455us) server_negotiation.cc:299] Negotiation successful
0810 07:25:22.015855 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":106,"threads_started":1}
I20260810 07:25:22.016239 20403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.010367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:37183 (local address 127.0.0.1:52696)
0810 07:25:22.010870 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:22.010883 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:22.010958 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:22.011782 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:22.011785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:22.011822 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:22.012028 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:22.012034 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.013075 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.013081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:22.014336 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:22.014349 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.015360 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.015363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:22.015512 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:22.015535 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 07:25:22.015605 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 07:25:22.015665 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":89,"spinlock_wait_cycles":11904,"thread_start_us":96,"threads_started":1}
I20260810 07:25:22.018680 19803 tablet_service.cc:1467] Processing CreateTablet for tablet f6c5489dce564788880d7731e678c458 (DEFAULT_TABLE table=test-table [id=8de85a31b72c48ca801e50fa8a12df1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:22.018761 20038 tablet_service.cc:1467] Processing CreateTablet for tablet f6c5489dce564788880d7731e678c458 (DEFAULT_TABLE table=test-table [id=8de85a31b72c48ca801e50fa8a12df1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:22.019668 19803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c5489dce564788880d7731e678c458. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:22.019668 20038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c5489dce564788880d7731e678c458. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:22.021694 20402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.009570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:34435 (local address 127.0.0.1:39136)
0810 07:25:22.010048 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:22.010120 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:22.010258 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:22.011787 (+ 1529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:22.011789 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:22.011824 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:22.012024 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:22.012032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.014416 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.014420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:22.015661 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:22.015675 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.021292 (+ 5617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.021295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:22.021401 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:22.021427 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 07:25:22.021482 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 07:25:22.021548 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":192,"spinlock_wait_cycles":3328,"thread_start_us":98,"threads_started":1}
I20260810 07:25:22.022367 20408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.010524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39136 (local address 127.12.157.195:34435)
0810 07:25:22.011209 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:22.011220 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:22.011239 (+ 19us) server_negotiation.cc:407] Connection header received
0810 07:25:22.011308 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:22.011313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:22.011602 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:22.011799 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:22.013122 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.014269 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.016373 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.017202 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.021912 (+ 4710us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:22.022050 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:22.022106 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 07:25:22.022201 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":166,"threads_started":1}
I20260810 07:25:22.025121 20262 tablet_service.cc:1467] Processing CreateTablet for tablet f6c5489dce564788880d7731e678c458 (DEFAULT_TABLE table=test-table [id=8de85a31b72c48ca801e50fa8a12df1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:22.026269 20262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6c5489dce564788880d7731e678c458. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:22.026660 20410 tablet_bootstrap.cc:492] T f6c5489dce564788880d7731e678c458 P e8a9b6d1d5334560b1ccf07ff07ea9a0: Bootstrap starting.
I20260810 07:25:22.027874 20411 tablet_bootstrap.cc:492] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740: Bootstrap starting.
I20260810 07:25:22.028852 20410 tablet_bootstrap.cc:654] T f6c5489dce564788880d7731e678c458 P e8a9b6d1d5334560b1ccf07ff07ea9a0: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:22.029637 20410 log.cc:824] T f6c5489dce564788880d7731e678c458 P e8a9b6d1d5334560b1ccf07ff07ea9a0: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:22.030418 20411 tablet_bootstrap.cc:654] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:22.031247 20411 log.cc:824] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:22.031796 20410 tablet_bootstrap.cc:492] T f6c5489dce564788880d7731e678c458 P e8a9b6d1d5334560b1ccf07ff07ea9a0: No bootstrap required, opened a new log
I20260810 07:25:22.032001 20410 ts_tablet_manager.cc:1397] T f6c5489dce564788880d7731e678c458 P e8a9b6d1d5334560b1ccf07ff07ea9a0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 07:25:22.033200 20411 tablet_bootstrap.cc:492] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740: No bootstrap required, opened a new log
I20260810 07:25:22.033403 20411 ts_tablet_manager.cc:1397] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 07:25:22.033869 20414 tablet_bootstrap.cc:492] T f6c5489dce564788880d7731e678c458 P 586ed486bb8b49fa9db378dda19d5125: Bootstrap starting.
I20260810 07:25:22.036155 20414 tablet_bootstrap.cc:654] T f6c5489dce564788880d7731e678c458 P 586ed486bb8b49fa9db378dda19d5125: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:22.036341 20411 raft_consensus.cc:348] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586ed486bb8b49fa9db378dda19d5125" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34435 } } peers { permanent_uuid: "e8a9b6d1d5334560b1ccf07ff07ea9a0" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 40943 } } peers { permanent_uuid: "83b66c627600491c92600d4ec1593740" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37183 } }
I20260810 07:25:22.036274 20410 raft_consensus.cc:348] T f6c5489dce564788880d7731e678c458 P e8a9b6d1d5334560b1ccf07ff07ea9a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586ed486bb8b49fa9db378dda19d5125" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34435 } } peers { permanent_uuid: "e8a9b6d1d5334560b1ccf07ff07ea9a0" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 40943 } } peers { permanent_uuid: "83b66c627600491c92600d4ec1593740" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37183 } }
I20260810 07:25:22.036581 20411 raft_consensus.cc:374] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:22.036583 20410 raft_consensus.cc:374] T f6c5489dce564788880d7731e678c458 P e8a9b6d1d5334560b1ccf07ff07ea9a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:22.036634 20411 raft_consensus.cc:729] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b66c627600491c92600d4ec1593740, State: Initialized, Role: FOLLOWER
I20260810 07:25:22.036638 20410 raft_consensus.cc:729] T f6c5489dce564788880d7731e678c458 P e8a9b6d1d5334560b1ccf07ff07ea9a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a9b6d1d5334560b1ccf07ff07ea9a0, State: Initialized, Role: FOLLOWER
I20260810 07:25:22.037102 20411 consensus_queue.cc:260] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586ed486bb8b49fa9db378dda19d5125" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34435 } } peers { permanent_uuid: "e8a9b6d1d5334560b1ccf07ff07ea9a0" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 40943 } } peers { permanent_uuid: "83b66c627600491c92600d4ec1593740" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37183 } }
I20260810 07:25:22.037134 20410 consensus_queue.cc:260] T f6c5489dce564788880d7731e678c458 P e8a9b6d1d5334560b1ccf07ff07ea9a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586ed486bb8b49fa9db378dda19d5125" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34435 } } peers { permanent_uuid: "e8a9b6d1d5334560b1ccf07ff07ea9a0" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 40943 } } peers { permanent_uuid: "83b66c627600491c92600d4ec1593740" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37183 } }
I20260810 07:25:22.037434 20414 log.cc:824] T f6c5489dce564788880d7731e678c458 P 586ed486bb8b49fa9db378dda19d5125: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:22.037809 20156 heartbeater.cc:503] Master 127.12.157.254:37371 was elected leader, sending a full tablet report...
I20260810 07:25:22.038092 20411 ts_tablet_manager.cc:1428] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260810 07:25:22.038148 20410 ts_tablet_manager.cc:1428] T f6c5489dce564788880d7731e678c458 P e8a9b6d1d5334560b1ccf07ff07ea9a0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 07:25:22.038563 19922 heartbeater.cc:503] Master 127.12.157.254:37371 was elected leader, sending a full tablet report...
I20260810 07:25:22.040278 20414 tablet_bootstrap.cc:492] T f6c5489dce564788880d7731e678c458 P 586ed486bb8b49fa9db378dda19d5125: No bootstrap required, opened a new log
I20260810 07:25:22.040475 20414 ts_tablet_manager.cc:1397] T f6c5489dce564788880d7731e678c458 P 586ed486bb8b49fa9db378dda19d5125: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 07:25:22.043802 20414 raft_consensus.cc:348] T f6c5489dce564788880d7731e678c458 P 586ed486bb8b49fa9db378dda19d5125 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586ed486bb8b49fa9db378dda19d5125" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34435 } } peers { permanent_uuid: "e8a9b6d1d5334560b1ccf07ff07ea9a0" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 40943 } } peers { permanent_uuid: "83b66c627600491c92600d4ec1593740" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37183 } }
I20260810 07:25:22.044049 20414 raft_consensus.cc:374] T f6c5489dce564788880d7731e678c458 P 586ed486bb8b49fa9db378dda19d5125 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:22.044106 20414 raft_consensus.cc:729] T f6c5489dce564788880d7731e678c458 P 586ed486bb8b49fa9db378dda19d5125 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 586ed486bb8b49fa9db378dda19d5125, State: Initialized, Role: FOLLOWER
I20260810 07:25:22.044493 20414 consensus_queue.cc:260] T f6c5489dce564788880d7731e678c458 P 586ed486bb8b49fa9db378dda19d5125 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586ed486bb8b49fa9db378dda19d5125" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34435 } } peers { permanent_uuid: "e8a9b6d1d5334560b1ccf07ff07ea9a0" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 40943 } } peers { permanent_uuid: "83b66c627600491c92600d4ec1593740" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37183 } }
I20260810 07:25:22.045127 20384 heartbeater.cc:503] Master 127.12.157.254:37371 was elected leader, sending a full tablet report...
I20260810 07:25:22.045393 20414 ts_tablet_manager.cc:1428] T f6c5489dce564788880d7731e678c458 P 586ed486bb8b49fa9db378dda19d5125: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 07:25:22.121694 20415 raft_consensus.cc:482] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:25:22.121920 20415 raft_consensus.cc:504] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586ed486bb8b49fa9db378dda19d5125" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34435 } } peers { permanent_uuid: "e8a9b6d1d5334560b1ccf07ff07ea9a0" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 40943 } } peers { permanent_uuid: "83b66c627600491c92600d4ec1593740" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37183 } }
I20260810 07:25:22.123172 20415 leader_election.cc:288] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 586ed486bb8b49fa9db378dda19d5125 (127.12.157.195:34435), e8a9b6d1d5334560b1ccf07ff07ea9a0 (127.12.157.193:40943)
I20260810 07:25:22.127497 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.123418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:34435 (local address 127.12.157.194:60811)
0810 07:25:22.123536 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:22.123553 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:22.123660 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:22.124118 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:22.124124 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:22.124170 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:22.124330 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:22.124338 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.125273 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.125277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:22.126213 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:22.126223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.127011 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.127018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:22.127130 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:22.127159 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 07:25:22.127245 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 07:25:22.127338 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"spinlock_wait_cycles":5376}
I20260810 07:25:22.127525 20408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.123691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:60811 (local address 127.12.157.195:34435)
0810 07:25:22.123858 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:22.123863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:22.123879 (+ 16us) server_negotiation.cc:407] Connection header received
0810 07:25:22.123937 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:22.123940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:22.123989 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:22.124075 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:22.124486 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.125130 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.126378 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.126878 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.126978 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:22.127022 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:22.127280 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 07:25:22.127389 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 07:25:22.127828 20404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.123646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:57431 (local address 127.12.157.193:40943)
0810 07:25:22.123787 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:22.123792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:22.124143 (+ 351us) server_negotiation.cc:407] Connection header received
0810 07:25:22.124283 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:22.124289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:22.124338 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:22.124432 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:22.124742 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.125461 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.126641 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.127238 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.127364 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:22.127411 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:22.127606 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 07:25:22.127693 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 07:25:22.127841 20420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.123530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:40943 (local address 127.12.157.194:57431)
0810 07:25:22.124037 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:22.124053 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:22.124165 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:22.124486 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:22.124489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:22.124508 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:22.124630 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:22.124637 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.125602 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.125605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:22.126485 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:22.126496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.127371 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.127374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:22.127468 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:22.127492 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 07:25:22.127576 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 07:25:22.127649 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":1,"spinlock_wait_cycles":12544,"thread_start_us":164,"threads_started":1}
I20260810 07:25:22.128311 20298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c5489dce564788880d7731e678c458" candidate_uuid: "83b66c627600491c92600d4ec1593740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "586ed486bb8b49fa9db378dda19d5125" is_pre_election: true
I20260810 07:25:22.128484 19817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c5489dce564788880d7731e678c458" candidate_uuid: "83b66c627600491c92600d4ec1593740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a9b6d1d5334560b1ccf07ff07ea9a0" is_pre_election: true
I20260810 07:25:22.128669 20298 raft_consensus.cc:2454] T f6c5489dce564788880d7731e678c458 P 586ed486bb8b49fa9db378dda19d5125 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b66c627600491c92600d4ec1593740 in term 0.
I20260810 07:25:22.128762 19817 raft_consensus.cc:2454] T f6c5489dce564788880d7731e678c458 P e8a9b6d1d5334560b1ccf07ff07ea9a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b66c627600491c92600d4ec1593740 in term 0.
I20260810 07:25:22.129225 19964 leader_election.cc:302] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 586ed486bb8b49fa9db378dda19d5125, 83b66c627600491c92600d4ec1593740; no voters:
I20260810 07:25:22.129532 20415 raft_consensus.cc:2781] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:25:22.129645 20415 raft_consensus.cc:482] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:25:22.129696 20415 raft_consensus.cc:3037] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:22.130887 20415 raft_consensus.cc:504] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586ed486bb8b49fa9db378dda19d5125" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34435 } } peers { permanent_uuid: "e8a9b6d1d5334560b1ccf07ff07ea9a0" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 40943 } } peers { permanent_uuid: "83b66c627600491c92600d4ec1593740" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37183 } }
I20260810 07:25:22.131297 20415 leader_election.cc:288] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740 [CANDIDATE]: Term 1 election: Requested vote from peers 586ed486bb8b49fa9db378dda19d5125 (127.12.157.195:34435), e8a9b6d1d5334560b1ccf07ff07ea9a0 (127.12.157.193:40943)
I20260810 07:25:22.131671 19817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c5489dce564788880d7731e678c458" candidate_uuid: "83b66c627600491c92600d4ec1593740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a9b6d1d5334560b1ccf07ff07ea9a0"
I20260810 07:25:22.131654 20298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6c5489dce564788880d7731e678c458" candidate_uuid: "83b66c627600491c92600d4ec1593740" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "586ed486bb8b49fa9db378dda19d5125"
I20260810 07:25:22.131793 19817 raft_consensus.cc:3037] T f6c5489dce564788880d7731e678c458 P e8a9b6d1d5334560b1ccf07ff07ea9a0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:22.131793 20298 raft_consensus.cc:3037] T f6c5489dce564788880d7731e678c458 P 586ed486bb8b49fa9db378dda19d5125 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:22.133198 20298 raft_consensus.cc:2454] T f6c5489dce564788880d7731e678c458 P 586ed486bb8b49fa9db378dda19d5125 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b66c627600491c92600d4ec1593740 in term 1.
I20260810 07:25:22.133198 19817 raft_consensus.cc:2454] T f6c5489dce564788880d7731e678c458 P e8a9b6d1d5334560b1ccf07ff07ea9a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b66c627600491c92600d4ec1593740 in term 1.
I20260810 07:25:22.133565 19965 leader_election.cc:302] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 586ed486bb8b49fa9db378dda19d5125, 83b66c627600491c92600d4ec1593740, e8a9b6d1d5334560b1ccf07ff07ea9a0; no voters:
I20260810 07:25:22.133749 20415 raft_consensus.cc:2781] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:22.134008 20415 raft_consensus.cc:686] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740 [term 1 LEADER]: Becoming Leader. State: Replica: 83b66c627600491c92600d4ec1593740, State: Running, Role: LEADER
I20260810 07:25:22.134368 20415 consensus_queue.cc:237] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586ed486bb8b49fa9db378dda19d5125" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34435 } } peers { permanent_uuid: "e8a9b6d1d5334560b1ccf07ff07ea9a0" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 40943 } } peers { permanent_uuid: "83b66c627600491c92600d4ec1593740" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37183 } }
I20260810 07:25:22.137812 19602 catalog_manager.cc:5495] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83b66c627600491c92600d4ec1593740 (127.12.157.194). New cstate: current_term: 1 leader_uuid: "83b66c627600491c92600d4ec1593740" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "586ed486bb8b49fa9db378dda19d5125" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8a9b6d1d5334560b1ccf07ff07ea9a0" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 40943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b66c627600491c92600d4ec1593740" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37183 } health_report { overall_health: HEALTHY } } }
I20260810 07:25:22.169107 20406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.164827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52698 (local address 127.12.157.194:37183)
0810 07:25:22.165022 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:22.165026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:22.165042 (+ 16us) server_negotiation.cc:407] Connection header received
0810 07:25:22.165122 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:22.165124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:22.165180 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:22.165278 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:25:22.165571 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.166111 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.167056 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:22.167283 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:22.167488 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:22.167735 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:25:22.168312 (+ 577us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:25:22.168790 (+ 478us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:22.168835 (+ 45us) server_negotiation.cc:299] Negotiation successful
0810 07:25:22.168899 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 07:25:22.175673 19817 raft_consensus.cc:1261] T f6c5489dce564788880d7731e678c458 P e8a9b6d1d5334560b1ccf07ff07ea9a0 [term 1 FOLLOWER]: Refusing update from remote peer 83b66c627600491c92600d4ec1593740: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:22.175725 20298 raft_consensus.cc:1261] T f6c5489dce564788880d7731e678c458 P 586ed486bb8b49fa9db378dda19d5125 [term 1 FOLLOWER]: Refusing update from remote peer 83b66c627600491c92600d4ec1593740: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:22.176482 20421 consensus_queue.cc:1035] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740 [LEADER]: Connected to new peer: Peer: permanent_uuid: "586ed486bb8b49fa9db378dda19d5125" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:22.176674 20415 consensus_queue.cc:1035] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8a9b6d1d5334560b1ccf07ff07ea9a0" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 40943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:22.186758 20432 mvcc.cc:205] Tried to move back new op lower bound from 7316876174023245824 to 7316876173865189376. Current Snapshot: MvccSnapshot[applied={T|T < 7316876174023245824}]
I20260810 07:25:22.197309 20429 mvcc.cc:205] Tried to move back new op lower bound from 7316876174023245824 to 7316876173865189376. Current Snapshot: MvccSnapshot[applied={T|T < 7316876174023245824}]
I20260810 07:25:22.202515 19602 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:22.202718 19602 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38290:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:22.205753 19602 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4086d063886b48c6923c1df7c9ae1df8: Sending DeleteTablet for 3 replicas of tablet f6c5489dce564788880d7731e678c458
I20260810 07:25:22.206736 20038 tablet_service.cc:1514] Processing DeleteTablet for tablet f6c5489dce564788880d7731e678c458 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:22 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:52696
I20260810 07:25:22.206753 19803 tablet_service.cc:1514] Processing DeleteTablet for tablet f6c5489dce564788880d7731e678c458 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:22 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:43466
I20260810 07:25:22.206938 20262 tablet_service.cc:1514] Processing DeleteTablet for tablet f6c5489dce564788880d7731e678c458 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:22 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:39136
I20260810 07:25:22.207356 20449 tablet_replica.cc:330] T f6c5489dce564788880d7731e678c458 P e8a9b6d1d5334560b1ccf07ff07ea9a0: stopping tablet replica
I20260810 07:25:22.207559 20450 tablet_replica.cc:330] T f6c5489dce564788880d7731e678c458 P 586ed486bb8b49fa9db378dda19d5125: stopping tablet replica
I20260810 07:25:22.207751 20449 raft_consensus.cc:2229] T f6c5489dce564788880d7731e678c458 P e8a9b6d1d5334560b1ccf07ff07ea9a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:25:22.208143 20449 raft_consensus.cc:2258] T f6c5489dce564788880d7731e678c458 P e8a9b6d1d5334560b1ccf07ff07ea9a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:22.208212 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 19709
I20260810 07:25:22.215226 20447 tablet_replica.cc:330] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740: stopping tablet replica
I20260810 07:25:22.215637 20447 raft_consensus.cc:2229] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:25:22.216262 20447 raft_consensus.cc:2258] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:22.219756 20447 ts_tablet_manager.cc:1905] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:22.223153 20447 ts_tablet_manager.cc:1918] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:25:22.223284 20447 log.cc:1196] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-1/wal/wals/f6c5489dce564788880d7731e678c458
I20260810 07:25:22.208125 20450 raft_consensus.cc:2229] T f6c5489dce564788880d7731e678c458 P 586ed486bb8b49fa9db378dda19d5125 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:25:22.223639 20447 ts_tablet_manager.cc:1939] T f6c5489dce564788880d7731e678c458 P 83b66c627600491c92600d4ec1593740: Deleting consensus metadata
I20260810 07:25:22.223923 20450 raft_consensus.cc:2258] T f6c5489dce564788880d7731e678c458 P 586ed486bb8b49fa9db378dda19d5125 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:22.225065 19584 catalog_manager.cc:4841] TS 83b66c627600491c92600d4ec1593740 (127.12.157.194:37183): tablet f6c5489dce564788880d7731e678c458 (table test-table [id=8de85a31b72c48ca801e50fa8a12df1b]) successfully deleted
I20260810 07:25:22.226223 20450 ts_tablet_manager.cc:1905] T f6c5489dce564788880d7731e678c458 P 586ed486bb8b49fa9db378dda19d5125: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:22.229969 20450 ts_tablet_manager.cc:1918] T f6c5489dce564788880d7731e678c458 P 586ed486bb8b49fa9db378dda19d5125: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:25:22.230134 20450 log.cc:1196] T f6c5489dce564788880d7731e678c458 P 586ed486bb8b49fa9db378dda19d5125: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786346694975122-12919-0/minicluster-data/ts-2/wal/wals/f6c5489dce564788880d7731e678c458
I20260810 07:25:22.230535 20450 ts_tablet_manager.cc:1939] T f6c5489dce564788880d7731e678c458 P 586ed486bb8b49fa9db378dda19d5125: Deleting consensus metadata
W20260810 07:25:22.231485 19585 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.157.193:40943 (error 108)
I20260810 07:25:22.231621 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 19931
I20260810 07:25:22.241353 20402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.240791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:40943 (local address 127.0.0.1:43482)
0810 07:25:22.241117 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:22.241215 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.193:40943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260810 07:25:22.243498 19585 catalog_manager.cc:4584] TS e8a9b6d1d5334560b1ccf07ff07ea9a0 (127.12.157.193:40943): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f6c5489dce564788880d7731e678c458: Network error: Client connection negotiation failed: client connection to 127.12.157.193:40943: connect: Connection refused (error 111)
I20260810 07:25:22.244837 19584 catalog_manager.cc:4841] TS 586ed486bb8b49fa9db378dda19d5125 (127.12.157.195:34435): tablet f6c5489dce564788880d7731e678c458 (table test-table [id=8de85a31b72c48ca801e50fa8a12df1b]) successfully deleted
I20260810 07:25:22.253944 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 20161
I20260810 07:25:22.271088 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 19557
2026-08-10T07:25:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2687 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T07:25:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:25:22Z Disabled control of system clock
I20260810 07:25:22.315614 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:40545
--webserver_interface=127.12.157.254
--webserver_port=0
--builtin_ntp_servers=127.12.157.212:45545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:40545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 07:25:22.430766 20461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:22.431039 20461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:22.431077 20461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:22.435038 20461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:25:22.435103 20461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:22.435119 20461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:22.435135 20461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:25:22.435150 20461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:25:22.439900 20461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:45545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:40545
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:40545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:22.440989 20461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:22.442340 20461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:22.451061 20467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:22.450992 20470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:22.451020 20468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:22.451110 20461 server_base.cc:1034] running on GCE node
I20260810 07:25:22.451921 20461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:22.453107 20461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:22.454272 20461 hybrid_clock.cc:648] HybridClock initialized: now 1786346722454248 us; error 43 us; skew 500 ppm
I20260810 07:25:22.456140 20461 webserver.cc:460] Webserver started at http://127.12.157.254:43213/ using document root <none> and password file <none>
I20260810 07:25:22.456724 20461 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:22.456789 20461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:22.457059 20461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:22.459129 20461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "bbcdd01cfb0746f68e33ba1912daeb84"
format_stamp: "Formatted at 2026-08-10 07:25:22 on dist-test-slave-ffrd"
server_key: "4166db9215bd0cefad1281152ddc1867"
server_key_iv: "c9404f82fd6288b241ea556b52f30564"
server_key_version: "encryptionkey@0"
I20260810 07:25:22.459712 20461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "bbcdd01cfb0746f68e33ba1912daeb84"
format_stamp: "Formatted at 2026-08-10 07:25:22 on dist-test-slave-ffrd"
server_key: "4166db9215bd0cefad1281152ddc1867"
server_key_iv: "c9404f82fd6288b241ea556b52f30564"
server_key_version: "encryptionkey@0"
I20260810 07:25:22.463558 20461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 07:25:22.466048 20478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:22.467300 20461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 07:25:22.467429 20461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "bbcdd01cfb0746f68e33ba1912daeb84"
format_stamp: "Formatted at 2026-08-10 07:25:22 on dist-test-slave-ffrd"
server_key: "4166db9215bd0cefad1281152ddc1867"
server_key_iv: "c9404f82fd6288b241ea556b52f30564"
server_key_version: "encryptionkey@0"
I20260810 07:25:22.467554 20461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:22.472394 20461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:22.473062 20461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:22.473258 20461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:22.480888 20461 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:40545
I20260810 07:25:22.480911 20576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:40545 every 8 connection(s)
I20260810 07:25:22.481918 20461 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:25:22.485311 20577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:22.491119 20577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbcdd01cfb0746f68e33ba1912daeb84: Bootstrap starting.
I20260810 07:25:22.491914 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 20461
I20260810 07:25:22.492051 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:25:22.492373 12919 external_mini_cluster.cc:1428] Setting key 6b4cf1b83f9726c58738ab3f07f6324d
I20260810 07:25:22.494328 20577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbcdd01cfb0746f68e33ba1912daeb84: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:22.495108 20577 log.cc:824] T 00000000000000000000000000000000 P bbcdd01cfb0746f68e33ba1912daeb84: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:22.497658 20577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbcdd01cfb0746f68e33ba1912daeb84: No bootstrap required, opened a new log
I20260810 07:25:22.498418 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.493844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49324 (local address 127.12.157.254:40545)
0810 07:25:22.494532 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:22.494549 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:22.494587 (+ 38us) server_negotiation.cc:407] Connection header received
0810 07:25:22.495306 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:22.495326 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:22.495770 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:22.496123 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:22.497079 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:22.497084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:22.497095 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:25:22.497107 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:22.497164 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:25:22.497180 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:22.497283 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:22.497432 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 07:25:22.497687 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":143,"threads_started":1}
I20260810 07:25:22.501348 20577 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbcdd01cfb0746f68e33ba1912daeb84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcdd01cfb0746f68e33ba1912daeb84" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 40545 } }
I20260810 07:25:22.501580 20577 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbcdd01cfb0746f68e33ba1912daeb84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:22.501614 20577 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbcdd01cfb0746f68e33ba1912daeb84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbcdd01cfb0746f68e33ba1912daeb84, State: Initialized, Role: FOLLOWER
I20260810 07:25:22.502157 20577 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbcdd01cfb0746f68e33ba1912daeb84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcdd01cfb0746f68e33ba1912daeb84" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 40545 } }
I20260810 07:25:22.502307 20577 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbcdd01cfb0746f68e33ba1912daeb84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:25:22.502377 20577 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbcdd01cfb0746f68e33ba1912daeb84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:25:22.502489 20577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbcdd01cfb0746f68e33ba1912daeb84 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:22.503491 20577 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbcdd01cfb0746f68e33ba1912daeb84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcdd01cfb0746f68e33ba1912daeb84" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 40545 } }
I20260810 07:25:22.503844 20577 leader_election.cc:302] T 00000000000000000000000000000000 P bbcdd01cfb0746f68e33ba1912daeb84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbcdd01cfb0746f68e33ba1912daeb84; no voters:
I20260810 07:25:22.504232 20577 leader_election.cc:288] T 00000000000000000000000000000000 P bbcdd01cfb0746f68e33ba1912daeb84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:25:22.504309 20594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbcdd01cfb0746f68e33ba1912daeb84 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:22.504534 20594 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbcdd01cfb0746f68e33ba1912daeb84 [term 1 LEADER]: Becoming Leader. State: Replica: bbcdd01cfb0746f68e33ba1912daeb84, State: Running, Role: LEADER
I20260810 07:25:22.504829 20594 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbcdd01cfb0746f68e33ba1912daeb84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcdd01cfb0746f68e33ba1912daeb84" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 40545 } }
I20260810 07:25:22.505640 20577 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbcdd01cfb0746f68e33ba1912daeb84 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:25:22.506798 20597 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbcdd01cfb0746f68e33ba1912daeb84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbcdd01cfb0746f68e33ba1912daeb84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcdd01cfb0746f68e33ba1912daeb84" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 40545 } } }
I20260810 07:25:22.506808 20596 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbcdd01cfb0746f68e33ba1912daeb84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbcdd01cfb0746f68e33ba1912daeb84. Latest consensus state: current_term: 1 leader_uuid: "bbcdd01cfb0746f68e33ba1912daeb84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcdd01cfb0746f68e33ba1912daeb84" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 40545 } } }
I20260810 07:25:22.506935 20597 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbcdd01cfb0746f68e33ba1912daeb84 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:22.506938 20596 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbcdd01cfb0746f68e33ba1912daeb84 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:22.507997 20607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:25:22.510780 20607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:25:22.516904 20607 catalog_manager.cc:1347] Generated new cluster ID: 848243d8f05049c49b966b73920b9e86
I20260810 07:25:22.517001 20607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:25:22.535077 20607 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:25:22.535434 20607 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:25:22.542006 20607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbcdd01cfb0746f68e33ba1912daeb84: Generated new TSK 0
I20260810 07:25:22.542795 20607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:25:22.554255 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:0
--local_ip_for_outbound_sockets=127.12.157.193
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:40545
--builtin_ntp_servers=127.12.157.212:45545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 07:25:22.681653 20625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:22.681941 20625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:22.682024 20625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:22.686164 20625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:22.686249 20625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:22.686338 20625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:25:22.691316 20625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:45545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:40545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:22.692495 20625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:22.693996 20625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:22.704319 20636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:22.704322 20633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:22.704322 20634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:22.704712 20625 server_base.cc:1034] running on GCE node
I20260810 07:25:22.705204 20625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:22.705922 20625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:22.707146 20625 hybrid_clock.cc:648] HybridClock initialized: now 1786346722707128 us; error 32 us; skew 500 ppm
I20260810 07:25:22.709095 20625 webserver.cc:460] Webserver started at http://127.12.157.193:41049/ using document root <none> and password file <none>
I20260810 07:25:22.709685 20625 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:22.709786 20625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:22.710022 20625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:22.712126 20625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-0/data/instance:
uuid: "3c7e1a58e302440d88d12b217956a9b5"
format_stamp: "Formatted at 2026-08-10 07:25:22 on dist-test-slave-ffrd"
server_key: "437d65b59a8553c73f3404af88bd5a45"
server_key_iv: "7c4964ba4dd9224e8f43a486f76e12c7"
server_key_version: "encryptionkey@0"
I20260810 07:25:22.712687 20625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance:
uuid: "3c7e1a58e302440d88d12b217956a9b5"
format_stamp: "Formatted at 2026-08-10 07:25:22 on dist-test-slave-ffrd"
server_key: "437d65b59a8553c73f3404af88bd5a45"
server_key_iv: "7c4964ba4dd9224e8f43a486f76e12c7"
server_key_version: "encryptionkey@0"
I20260810 07:25:22.716513 20625 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260810 07:25:22.719239 20647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:22.720463 20625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 07:25:22.720606 20625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "3c7e1a58e302440d88d12b217956a9b5"
format_stamp: "Formatted at 2026-08-10 07:25:22 on dist-test-slave-ffrd"
server_key: "437d65b59a8553c73f3404af88bd5a45"
server_key_iv: "7c4964ba4dd9224e8f43a486f76e12c7"
server_key_version: "encryptionkey@0"
I20260810 07:25:22.720727 20625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:22.725448 20625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:22.726083 20625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:22.726270 20625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:22.726833 20625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:22.727766 20625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:22.727844 20625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:22.727916 20625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:22.727957 20625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:22.737175 20625 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:37863
I20260810 07:25:22.737224 20852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:37863 every 8 connection(s)
I20260810 07:25:22.738107 20625 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:25:22.742568 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 20625
I20260810 07:25:22.742705 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance
I20260810 07:25:22.742993 12919 external_mini_cluster.cc:1428] Setting key 69574f9fb0af79ed151e2e85a297706f
I20260810 07:25:22.744439 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.739930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:54001 (local address 127.12.157.254:40545)
0810 07:25:22.740074 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:22.740078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:22.741206 (+ 1128us) server_negotiation.cc:407] Connection header received
0810 07:25:22.742579 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:22.742583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:22.742630 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:22.742712 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:22.743976 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:22.743980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:22.743982 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:25:22.743995 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:22.744018 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:25:22.744029 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:22.744118 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:22.744249 (+ 131us) server_negotiation.cc:299] Negotiation successful
0810 07:25:22.744293 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 07:25:22.745172 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:0
--local_ip_for_outbound_sockets=127.12.157.194
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:40545
--builtin_ntp_servers=127.12.157.212:45545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 07:25:22.745426 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.740331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:40545 (local address 127.12.157.193:54001)
0810 07:25:22.741069 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:22.741110 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:22.742369 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:22.742908 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:22.742914 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:22.743002 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:22.743674 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:25:22.743676 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:22.743730 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:25:22.743816 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:22.744118 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:22.744126 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:22.744130 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:25:22.744261 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 07:25:22.744545 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":120,"threads_started":1}
I20260810 07:25:22.746610 20853 heartbeater.cc:347] Connected to a master server at 127.12.157.254:40545
I20260810 07:25:22.746855 20853 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:22.747267 20853 heartbeater.cc:511] Master 127.12.157.254:40545 requested a full tablet report, sending...
I20260810 07:25:22.748771 20505 ts_manager.cc:194] Registered new tserver with Master: 3c7e1a58e302440d88d12b217956a9b5 (127.12.157.193:37863)
W20260810 07:25:22.863204 20856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:22.863464 20856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:22.863504 20856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:22.867507 20856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:22.867561 20856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:22.867645 20856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:25:22.872511 20856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:45545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:40545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:22.873605 20856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:22.875213 20856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:22.885618 20861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:22.885643 20866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:22.885632 20862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:22.886030 20856 server_base.cc:1034] running on GCE node
I20260810 07:25:22.886488 20856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:22.887146 20856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:22.888336 20856 hybrid_clock.cc:648] HybridClock initialized: now 1786346722888314 us; error 33 us; skew 500 ppm
I20260810 07:25:22.890376 20856 webserver.cc:460] Webserver started at http://127.12.157.194:39327/ using document root <none> and password file <none>
I20260810 07:25:22.890970 20856 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:22.891065 20856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:22.891304 20856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:22.893543 20856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-1/data/instance:
uuid: "5a7ebdfc6bc04b7395b454a41a7eec64"
format_stamp: "Formatted at 2026-08-10 07:25:22 on dist-test-slave-ffrd"
server_key: "e7485b56687781f1f97f7704591a162f"
server_key_iv: "f6d2292856a5cd8fb5104c49bc96919c"
server_key_version: "encryptionkey@0"
I20260810 07:25:22.894135 20856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance:
uuid: "5a7ebdfc6bc04b7395b454a41a7eec64"
format_stamp: "Formatted at 2026-08-10 07:25:22 on dist-test-slave-ffrd"
server_key: "e7485b56687781f1f97f7704591a162f"
server_key_iv: "f6d2292856a5cd8fb5104c49bc96919c"
server_key_version: "encryptionkey@0"
I20260810 07:25:22.898130 20856 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 07:25:22.900911 20877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:22.902354 20856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:25:22.902513 20856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "5a7ebdfc6bc04b7395b454a41a7eec64"
format_stamp: "Formatted at 2026-08-10 07:25:22 on dist-test-slave-ffrd"
server_key: "e7485b56687781f1f97f7704591a162f"
server_key_iv: "f6d2292856a5cd8fb5104c49bc96919c"
server_key_version: "encryptionkey@0"
I20260810 07:25:22.902647 20856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:22.907794 20856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:22.908442 20856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:22.908634 20856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:22.909282 20856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:22.910532 20856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:22.910601 20856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:22.910657 20856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:22.910694 20856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:22.920222 20856 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:33719
I20260810 07:25:22.920244 21083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:33719 every 8 connection(s)
I20260810 07:25:22.921205 20856 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:25:22.923199 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 20856
I20260810 07:25:22.923347 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance
I20260810 07:25:22.923702 12919 external_mini_cluster.cc:1428] Setting key cd62717c425dabdbd3555d2e73303c05
I20260810 07:25:22.926086 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:0
--local_ip_for_outbound_sockets=127.12.157.195
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:40545
--builtin_ntp_servers=127.12.157.212:45545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 07:25:22.928251 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.923082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:46913 (local address 127.12.157.254:40545)
0810 07:25:22.923239 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:22.923244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:22.925030 (+ 1786us) server_negotiation.cc:407] Connection header received
0810 07:25:22.926363 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:22.926365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:22.926401 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:22.926459 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:22.927664 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:22.927667 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:22.927669 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:25:22.927680 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:22.927700 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:25:22.927707 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:22.927790 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:22.928016 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 07:25:22.928073 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 07:25:22.929257 21088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:22.923547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:40545 (local address 127.12.157.194:46913)
0810 07:25:22.924348 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:22.924411 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:22.926098 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:22.926634 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:22.926646 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:22.926760 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:22.927344 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:25:22.927348 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:22.927413 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:25:22.927506 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:22.927793 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:22.927804 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:22.927813 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 07:25:22.928000 (+ 187us) client_negotiation.cc:240] Negotiation successful
0810 07:25:22.928260 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":147,"threads_started":1}
I20260810 07:25:22.930444 21084 heartbeater.cc:347] Connected to a master server at 127.12.157.254:40545
I20260810 07:25:22.930656 21084 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:22.931097 21084 heartbeater.cc:511] Master 127.12.157.254:40545 requested a full tablet report, sending...
I20260810 07:25:22.932126 20505 ts_manager.cc:194] Registered new tserver with Master: 5a7ebdfc6bc04b7395b454a41a7eec64 (127.12.157.194:33719)
W20260810 07:25:23.045046 21089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:23.045377 21089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:23.045475 21089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:23.050010 21089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:23.050107 21089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:23.050197 21089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:25:23.055384 21089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:45545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:40545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:23.056543 21089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:23.058060 21089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:23.067807 21097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:23.067787 21096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:23.067781 21104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:23.067924 21089 server_base.cc:1034] running on GCE node
I20260810 07:25:23.068500 21089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:23.069190 21089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:23.070351 21089 hybrid_clock.cc:648] HybridClock initialized: now 1786346723070324 us; error 40 us; skew 500 ppm
I20260810 07:25:23.072212 21089 webserver.cc:460] Webserver started at http://127.12.157.195:38841/ using document root <none> and password file <none>
I20260810 07:25:23.072790 21089 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:23.072872 21089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:23.073128 21089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:23.075137 21089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-2/data/instance:
uuid: "303621182f8e4f47ae9a8c63445914d2"
format_stamp: "Formatted at 2026-08-10 07:25:23 on dist-test-slave-ffrd"
server_key: "42a6d44840c7e99421e9a720242eeca2"
server_key_iv: "d25e73bc69d20b3cbbe2c58cffe430a8"
server_key_version: "encryptionkey@0"
I20260810 07:25:23.075701 21089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance:
uuid: "303621182f8e4f47ae9a8c63445914d2"
format_stamp: "Formatted at 2026-08-10 07:25:23 on dist-test-slave-ffrd"
server_key: "42a6d44840c7e99421e9a720242eeca2"
server_key_iv: "d25e73bc69d20b3cbbe2c58cffe430a8"
server_key_version: "encryptionkey@0"
I20260810 07:25:23.079402 21089 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 07:25:23.082103 21111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:23.083271 21089 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 07:25:23.083397 21089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "303621182f8e4f47ae9a8c63445914d2"
format_stamp: "Formatted at 2026-08-10 07:25:23 on dist-test-slave-ffrd"
server_key: "42a6d44840c7e99421e9a720242eeca2"
server_key_iv: "d25e73bc69d20b3cbbe2c58cffe430a8"
server_key_version: "encryptionkey@0"
I20260810 07:25:23.083500 21089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:23.087983 21089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:23.088585 21089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:23.088762 21089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:23.089380 21089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:23.090293 21089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:23.090340 21089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:23.090440 21089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:23.090478 21089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:23.099797 21089 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:46001
I20260810 07:25:23.099817 21325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:46001 every 8 connection(s)
I20260810 07:25:23.100719 21089 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:25:23.104353 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 21089
I20260810 07:25:23.104486 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance
I20260810 07:25:23.104800 12919 external_mini_cluster.cc:1428] Setting key 688cfe626aedc3be0bc38d0a0e04c688
I20260810 07:25:23.107722 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:23.102743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:48541 (local address 127.12.157.254:40545)
0810 07:25:23.102876 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:23.102885 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:23.103933 (+ 1048us) server_negotiation.cc:407] Connection header received
0810 07:25:23.105819 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:23.105823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:23.105872 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:23.105984 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:23.107276 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:23.107279 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:23.107281 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:25:23.107293 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:23.107315 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:25:23.107323 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:23.107402 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:23.107588 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 07:25:23.107622 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 07:25:23.108649 21329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:23.103091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:40545 (local address 127.12.157.195:48541)
0810 07:25:23.103764 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:23.103819 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:23.105547 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:23.106187 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:23.106198 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:23.106309 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:23.106968 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:25:23.106971 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:23.107028 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:25:23.107122 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:23.107446 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:23.107456 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:23.107461 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:25:23.107628 (+ 167us) client_negotiation.cc:240] Negotiation successful
0810 07:25:23.107859 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":117,"threads_started":1}
I20260810 07:25:23.109741 21326 heartbeater.cc:347] Connected to a master server at 127.12.157.254:40545
I20260810 07:25:23.109993 21326 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:23.110414 21326 heartbeater.cc:511] Master 127.12.157.254:40545 requested a full tablet report, sending...
I20260810 07:25:23.111454 20505 ts_manager.cc:194] Registered new tserver with Master: 303621182f8e4f47ae9a8c63445914d2 (127.12.157.195:46001)
I20260810 07:25:23.118896 12919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:25:23.122346 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:23.120818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49334 (local address 127.12.157.254:40545)
0810 07:25:23.120992 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:23.121000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:23.121152 (+ 152us) server_negotiation.cc:407] Connection header received
0810 07:25:23.121255 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:23.121257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:23.121290 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:23.121361 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:23.122049 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:23.122052 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:23.122054 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:25:23.122065 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:23.122083 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:25:23.122090 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:23.122160 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:23.122209 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 07:25:23.122242 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 07:25:23.125202 20505 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49334:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:25:23.128270 20505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:25:23.140970 21353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:23.138225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49772 (local address 127.12.157.193:37863)
0810 07:25:23.138963 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:23.138975 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:23.138999 (+ 24us) server_negotiation.cc:407] Connection header received
0810 07:25:23.139052 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:23.139056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:23.139217 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:23.139355 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:23.140447 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:23.140455 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:23.140464 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:25:23.140474 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:23.140510 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:25:23.140538 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:23.140640 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:23.140729 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 07:25:23.140810 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":103,"threads_started":1}
I20260810 07:25:23.140930 21349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:23.137249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:33719 (local address 127.0.0.1:39142)
0810 07:25:23.137672 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:23.137730 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:23.137852 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:23.139221 (+ 1369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:23.139227 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:23.139318 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:23.140034 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:25:23.140036 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:23.140048 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:25:23.140092 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:23.140548 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:23.140559 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:23.140569 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 07:25:23.140680 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 07:25:23.140753 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":125,"threads_started":1}
I20260810 07:25:23.140990 21352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:23.138039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:37863 (local address 127.0.0.1:49772)
0810 07:25:23.138298 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:23.138313 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:23.138394 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:23.139361 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:23.139365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:23.139380 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:23.139993 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:25:23.139996 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:23.140019 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:25:23.140092 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:23.140618 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:23.140621 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:23.140625 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:25:23.140720 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 07:25:23.140759 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":108,"threads_started":1}
I20260810 07:25:23.141209 21355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:23.138366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:46001 (local address 127.0.0.1:48606)
0810 07:25:23.138608 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:23.138622 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:23.138695 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:23.140132 (+ 1437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:23.140138 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:23.140157 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:23.140591 (+ 434us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:25:23.140593 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:23.140605 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:25:23.140629 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:23.141004 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:23.141011 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:23.141016 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:25:23.141075 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 07:25:23.141111 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":100,"threads_started":1}
I20260810 07:25:23.141690 21356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:23.138725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48606 (local address 127.12.157.195:46001)
0810 07:25:23.139188 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:23.139197 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:23.139215 (+ 18us) server_negotiation.cc:407] Connection header received
0810 07:25:23.139258 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:23.139261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:23.139431 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:23.139603 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:23.140765 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:23.140771 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:23.140784 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:25:23.140797 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:23.140840 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:25:23.140872 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:23.141393 (+ 521us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:23.141460 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 07:25:23.141530 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":162,"threads_started":1}
I20260810 07:25:23.141741 21351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:23.137466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39142 (local address 127.12.157.194:33719)
0810 07:25:23.138007 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:23.138032 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:23.138052 (+ 20us) server_negotiation.cc:407] Connection header received
0810 07:25:23.138119 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:23.138127 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:23.138342 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:23.138492 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:23.140260 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:23.140267 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:23.140281 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:25:23.140293 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:23.140355 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:25:23.140390 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:23.140501 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:23.141505 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0810 07:25:23.141578 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":198,"threads_started":1}
I20260810 07:25:23.144235 20737 tablet_service.cc:1467] Processing CreateTablet for tablet f362b4b0bc2248ea80dcaca8c6d627ef (DEFAULT_TABLE table=test-table [id=bf099b1d96e94ce9abd922e272068605]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:23.144346 21210 tablet_service.cc:1467] Processing CreateTablet for tablet f362b4b0bc2248ea80dcaca8c6d627ef (DEFAULT_TABLE table=test-table [id=bf099b1d96e94ce9abd922e272068605]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:23.144346 20963 tablet_service.cc:1467] Processing CreateTablet for tablet f362b4b0bc2248ea80dcaca8c6d627ef (DEFAULT_TABLE table=test-table [id=bf099b1d96e94ce9abd922e272068605]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:23.145370 20737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f362b4b0bc2248ea80dcaca8c6d627ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:23.145431 21210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f362b4b0bc2248ea80dcaca8c6d627ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:23.145431 20963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f362b4b0bc2248ea80dcaca8c6d627ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:23.152349 21357 tablet_bootstrap.cc:492] T f362b4b0bc2248ea80dcaca8c6d627ef P 3c7e1a58e302440d88d12b217956a9b5: Bootstrap starting.
I20260810 07:25:23.154069 21359 tablet_bootstrap.cc:492] T f362b4b0bc2248ea80dcaca8c6d627ef P 303621182f8e4f47ae9a8c63445914d2: Bootstrap starting.
I20260810 07:25:23.155329 21357 tablet_bootstrap.cc:654] T f362b4b0bc2248ea80dcaca8c6d627ef P 3c7e1a58e302440d88d12b217956a9b5: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:23.155588 21358 tablet_bootstrap.cc:492] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64: Bootstrap starting.
I20260810 07:25:23.156322 21359 tablet_bootstrap.cc:654] T f362b4b0bc2248ea80dcaca8c6d627ef P 303621182f8e4f47ae9a8c63445914d2: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:23.156422 21357 log.cc:824] T f362b4b0bc2248ea80dcaca8c6d627ef P 3c7e1a58e302440d88d12b217956a9b5: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:23.157234 21359 log.cc:824] T f362b4b0bc2248ea80dcaca8c6d627ef P 303621182f8e4f47ae9a8c63445914d2: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:23.157804 21358 tablet_bootstrap.cc:654] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:23.158506 21358 log.cc:824] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:23.158744 21357 tablet_bootstrap.cc:492] T f362b4b0bc2248ea80dcaca8c6d627ef P 3c7e1a58e302440d88d12b217956a9b5: No bootstrap required, opened a new log
I20260810 07:25:23.158962 21357 ts_tablet_manager.cc:1397] T f362b4b0bc2248ea80dcaca8c6d627ef P 3c7e1a58e302440d88d12b217956a9b5: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 07:25:23.159205 21359 tablet_bootstrap.cc:492] T f362b4b0bc2248ea80dcaca8c6d627ef P 303621182f8e4f47ae9a8c63445914d2: No bootstrap required, opened a new log
I20260810 07:25:23.159389 21359 ts_tablet_manager.cc:1397] T f362b4b0bc2248ea80dcaca8c6d627ef P 303621182f8e4f47ae9a8c63445914d2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 07:25:23.160236 21358 tablet_bootstrap.cc:492] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64: No bootstrap required, opened a new log
I20260810 07:25:23.160430 21358 ts_tablet_manager.cc:1397] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 07:25:23.163884 21357 raft_consensus.cc:348] T f362b4b0bc2248ea80dcaca8c6d627ef P 3c7e1a58e302440d88d12b217956a9b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a7ebdfc6bc04b7395b454a41a7eec64" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 33719 } } peers { permanent_uuid: "3c7e1a58e302440d88d12b217956a9b5" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 37863 } } peers { permanent_uuid: "303621182f8e4f47ae9a8c63445914d2" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 46001 } }
I20260810 07:25:23.163884 21359 raft_consensus.cc:348] T f362b4b0bc2248ea80dcaca8c6d627ef P 303621182f8e4f47ae9a8c63445914d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a7ebdfc6bc04b7395b454a41a7eec64" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 33719 } } peers { permanent_uuid: "3c7e1a58e302440d88d12b217956a9b5" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 37863 } } peers { permanent_uuid: "303621182f8e4f47ae9a8c63445914d2" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 46001 } }
I20260810 07:25:23.164171 21359 raft_consensus.cc:374] T f362b4b0bc2248ea80dcaca8c6d627ef P 303621182f8e4f47ae9a8c63445914d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:23.164171 21357 raft_consensus.cc:374] T f362b4b0bc2248ea80dcaca8c6d627ef P 3c7e1a58e302440d88d12b217956a9b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:23.164240 21359 raft_consensus.cc:729] T f362b4b0bc2248ea80dcaca8c6d627ef P 303621182f8e4f47ae9a8c63445914d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 303621182f8e4f47ae9a8c63445914d2, State: Initialized, Role: FOLLOWER
I20260810 07:25:23.164060 21358 raft_consensus.cc:348] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a7ebdfc6bc04b7395b454a41a7eec64" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 33719 } } peers { permanent_uuid: "3c7e1a58e302440d88d12b217956a9b5" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 37863 } } peers { permanent_uuid: "303621182f8e4f47ae9a8c63445914d2" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 46001 } }
I20260810 07:25:23.164240 21357 raft_consensus.cc:729] T f362b4b0bc2248ea80dcaca8c6d627ef P 3c7e1a58e302440d88d12b217956a9b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c7e1a58e302440d88d12b217956a9b5, State: Initialized, Role: FOLLOWER
I20260810 07:25:23.164309 21358 raft_consensus.cc:374] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:23.164381 21358 raft_consensus.cc:729] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a7ebdfc6bc04b7395b454a41a7eec64, State: Initialized, Role: FOLLOWER
I20260810 07:25:23.164732 21359 consensus_queue.cc:260] T f362b4b0bc2248ea80dcaca8c6d627ef P 303621182f8e4f47ae9a8c63445914d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a7ebdfc6bc04b7395b454a41a7eec64" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 33719 } } peers { permanent_uuid: "3c7e1a58e302440d88d12b217956a9b5" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 37863 } } peers { permanent_uuid: "303621182f8e4f47ae9a8c63445914d2" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 46001 } }
I20260810 07:25:23.164719 21357 consensus_queue.cc:260] T f362b4b0bc2248ea80dcaca8c6d627ef P 3c7e1a58e302440d88d12b217956a9b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a7ebdfc6bc04b7395b454a41a7eec64" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 33719 } } peers { permanent_uuid: "3c7e1a58e302440d88d12b217956a9b5" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 37863 } } peers { permanent_uuid: "303621182f8e4f47ae9a8c63445914d2" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 46001 } }
I20260810 07:25:23.164795 21358 consensus_queue.cc:260] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a7ebdfc6bc04b7395b454a41a7eec64" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 33719 } } peers { permanent_uuid: "3c7e1a58e302440d88d12b217956a9b5" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 37863 } } peers { permanent_uuid: "303621182f8e4f47ae9a8c63445914d2" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 46001 } }
I20260810 07:25:23.165472 20853 heartbeater.cc:503] Master 127.12.157.254:40545 was elected leader, sending a full tablet report...
I20260810 07:25:23.165702 21326 heartbeater.cc:503] Master 127.12.157.254:40545 was elected leader, sending a full tablet report...
I20260810 07:25:23.165792 21357 ts_tablet_manager.cc:1428] T f362b4b0bc2248ea80dcaca8c6d627ef P 3c7e1a58e302440d88d12b217956a9b5: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 07:25:23.166330 21084 heartbeater.cc:503] Master 127.12.157.254:40545 was elected leader, sending a full tablet report...
I20260810 07:25:23.166452 21359 ts_tablet_manager.cc:1428] T f362b4b0bc2248ea80dcaca8c6d627ef P 303621182f8e4f47ae9a8c63445914d2: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 07:25:23.167191 21358 ts_tablet_manager.cc:1428] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 07:25:23.168044 21375 raft_consensus.cc:482] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:25:23.168186 21375 raft_consensus.cc:504] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a7ebdfc6bc04b7395b454a41a7eec64" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 33719 } } peers { permanent_uuid: "3c7e1a58e302440d88d12b217956a9b5" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 37863 } } peers { permanent_uuid: "303621182f8e4f47ae9a8c63445914d2" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 46001 } }
I20260810 07:25:23.170306 21375 leader_election.cc:288] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c7e1a58e302440d88d12b217956a9b5 (127.12.157.193:37863), 303621182f8e4f47ae9a8c63445914d2 (127.12.157.195:46001)
I20260810 07:25:23.171943 21088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:23.169581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:37863 (local address 127.12.157.194:54125)
0810 07:25:23.169807 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:23.169825 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:23.169908 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:23.170632 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:23.170637 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:23.170655 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:23.171276 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:25:23.171280 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:23.171292 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:25:23.171320 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:23.171658 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:23.171662 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:23.171666 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:25:23.171741 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 07:25:23.171789 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
W20260810 07:25:23.172184 21086 tablet.cc:2369] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:25:23.172266 21353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:23.169670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:54125 (local address 127.12.157.193:37863)
0810 07:25:23.170080 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:23.170087 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:23.170105 (+ 18us) server_negotiation.cc:407] Connection header received
0810 07:25:23.170146 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:23.170149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:23.170179 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:23.170255 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:23.171455 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:23.171459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:23.171461 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:25:23.171475 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:23.171498 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:25:23.171509 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:23.171609 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:23.172099 (+ 490us) server_negotiation.cc:299] Negotiation successful
0810 07:25:23.172139 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322}
I20260810 07:25:23.172577 21356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:23.170405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:59135 (local address 127.12.157.195:46001)
0810 07:25:23.170532 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:23.170537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:23.170926 (+ 389us) server_negotiation.cc:407] Connection header received
0810 07:25:23.171104 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:23.171109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:23.171145 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:23.171198 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:23.172140 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:23.172143 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:23.172145 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:25:23.172158 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:23.172179 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:25:23.172190 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:23.172256 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:23.172429 (+ 173us) server_negotiation.cc:299] Negotiation successful
0810 07:25:23.172470 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 07:25:23.172786 20773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f362b4b0bc2248ea80dcaca8c6d627ef" candidate_uuid: "5a7ebdfc6bc04b7395b454a41a7eec64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c7e1a58e302440d88d12b217956a9b5" is_pre_election: true
I20260810 07:25:23.172922 21377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:23.170068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:46001 (local address 127.12.157.194:59135)
0810 07:25:23.170826 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:23.170843 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:23.170983 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:23.171329 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:23.171331 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:23.171349 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:23.171965 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:25:23.171967 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:23.171979 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:25:23.172006 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:23.172328 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:23.172331 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:23.172333 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 07:25:23.172722 (+ 389us) client_negotiation.cc:240] Negotiation successful
0810 07:25:23.172781 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":105,"threads_started":1}
I20260810 07:25:23.173092 20773 raft_consensus.cc:2454] T f362b4b0bc2248ea80dcaca8c6d627ef P 3c7e1a58e302440d88d12b217956a9b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a7ebdfc6bc04b7395b454a41a7eec64 in term 0.
I20260810 07:25:23.173457 21246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f362b4b0bc2248ea80dcaca8c6d627ef" candidate_uuid: "5a7ebdfc6bc04b7395b454a41a7eec64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "303621182f8e4f47ae9a8c63445914d2" is_pre_election: true
I20260810 07:25:23.173523 20881 leader_election.cc:302] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c7e1a58e302440d88d12b217956a9b5, 5a7ebdfc6bc04b7395b454a41a7eec64; no voters:
I20260810 07:25:23.173676 21246 raft_consensus.cc:2454] T f362b4b0bc2248ea80dcaca8c6d627ef P 303621182f8e4f47ae9a8c63445914d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a7ebdfc6bc04b7395b454a41a7eec64 in term 0.
I20260810 07:25:23.173808 21375 raft_consensus.cc:2781] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:25:23.173894 21375 raft_consensus.cc:482] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:25:23.173954 21375 raft_consensus.cc:3037] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:23.174996 21375 raft_consensus.cc:504] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a7ebdfc6bc04b7395b454a41a7eec64" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 33719 } } peers { permanent_uuid: "3c7e1a58e302440d88d12b217956a9b5" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 37863 } } peers { permanent_uuid: "303621182f8e4f47ae9a8c63445914d2" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 46001 } }
I20260810 07:25:23.175310 21375 leader_election.cc:288] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64 [CANDIDATE]: Term 1 election: Requested vote from peers 3c7e1a58e302440d88d12b217956a9b5 (127.12.157.193:37863), 303621182f8e4f47ae9a8c63445914d2 (127.12.157.195:46001)
I20260810 07:25:23.175661 20773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f362b4b0bc2248ea80dcaca8c6d627ef" candidate_uuid: "5a7ebdfc6bc04b7395b454a41a7eec64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c7e1a58e302440d88d12b217956a9b5"
I20260810 07:25:23.175742 21246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f362b4b0bc2248ea80dcaca8c6d627ef" candidate_uuid: "5a7ebdfc6bc04b7395b454a41a7eec64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "303621182f8e4f47ae9a8c63445914d2"
I20260810 07:25:23.175800 20773 raft_consensus.cc:3037] T f362b4b0bc2248ea80dcaca8c6d627ef P 3c7e1a58e302440d88d12b217956a9b5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:23.175877 21246 raft_consensus.cc:3037] T f362b4b0bc2248ea80dcaca8c6d627ef P 303621182f8e4f47ae9a8c63445914d2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:23.176998 20773 raft_consensus.cc:2454] T f362b4b0bc2248ea80dcaca8c6d627ef P 3c7e1a58e302440d88d12b217956a9b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a7ebdfc6bc04b7395b454a41a7eec64 in term 1.
I20260810 07:25:23.177395 21246 raft_consensus.cc:2454] T f362b4b0bc2248ea80dcaca8c6d627ef P 303621182f8e4f47ae9a8c63445914d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a7ebdfc6bc04b7395b454a41a7eec64 in term 1.
I20260810 07:25:23.177696 20881 leader_election.cc:302] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 303621182f8e4f47ae9a8c63445914d2, 5a7ebdfc6bc04b7395b454a41a7eec64; no voters:
I20260810 07:25:23.177881 21375 raft_consensus.cc:2781] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:23.178115 21375 raft_consensus.cc:686] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64 [term 1 LEADER]: Becoming Leader. State: Replica: 5a7ebdfc6bc04b7395b454a41a7eec64, State: Running, Role: LEADER
I20260810 07:25:23.178431 21375 consensus_queue.cc:237] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a7ebdfc6bc04b7395b454a41a7eec64" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 33719 } } peers { permanent_uuid: "3c7e1a58e302440d88d12b217956a9b5" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 37863 } } peers { permanent_uuid: "303621182f8e4f47ae9a8c63445914d2" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 46001 } }
I20260810 07:25:23.181591 20503 catalog_manager.cc:5495] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a7ebdfc6bc04b7395b454a41a7eec64 (127.12.157.194). New cstate: current_term: 1 leader_uuid: "5a7ebdfc6bc04b7395b454a41a7eec64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a7ebdfc6bc04b7395b454a41a7eec64" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 33719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c7e1a58e302440d88d12b217956a9b5" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 37863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "303621182f8e4f47ae9a8c63445914d2" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 46001 } health_report { overall_health: UNKNOWN } } }
I20260810 07:25:23.194430 21351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:23.192522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39158 (local address 127.12.157.194:33719)
0810 07:25:23.192679 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:23.192684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:23.192699 (+ 15us) server_negotiation.cc:407] Connection header received
0810 07:25:23.192865 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:23.192869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:23.192987 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:23.193085 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:23.193979 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:23.193983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:23.193986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:25:23.194002 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:23.194023 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:25:23.194052 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:23.194151 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:23.194236 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 07:25:23.194279 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 07:25:23.200531 21246 raft_consensus.cc:1261] T f362b4b0bc2248ea80dcaca8c6d627ef P 303621182f8e4f47ae9a8c63445914d2 [term 1 FOLLOWER]: Refusing update from remote peer 5a7ebdfc6bc04b7395b454a41a7eec64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:23.200546 20773 raft_consensus.cc:1261] T f362b4b0bc2248ea80dcaca8c6d627ef P 3c7e1a58e302440d88d12b217956a9b5 [term 1 FOLLOWER]: Refusing update from remote peer 5a7ebdfc6bc04b7395b454a41a7eec64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:23.201296 21375 consensus_queue.cc:1035] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "303621182f8e4f47ae9a8c63445914d2" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 46001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:23.201578 21387 consensus_queue.cc:1035] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c7e1a58e302440d88d12b217956a9b5" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 37863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:23.208287 21392 mvcc.cc:205] Tried to move back new op lower bound from 7316876178221850624 to 7316876178141159424. Current Snapshot: MvccSnapshot[applied={T|T < 7316876178221850624}]
I20260810 07:25:23.211802 21397 mvcc.cc:205] Tried to move back new op lower bound from 7316876178221850624 to 7316876178141159424. Current Snapshot: MvccSnapshot[applied={T|T < 7316876178221850624}]
I20260810 07:25:23.217630 20503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:23.217821 20503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49334:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:23.220461 20503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bbcdd01cfb0746f68e33ba1912daeb84: Sending DeleteTablet for 3 replicas of tablet f362b4b0bc2248ea80dcaca8c6d627ef
I20260810 07:25:23.221279 20963 tablet_service.cc:1514] Processing DeleteTablet for tablet f362b4b0bc2248ea80dcaca8c6d627ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:23 UTC) from {username='slave'} at 127.0.0.1:39142
I20260810 07:25:23.221465 20737 tablet_service.cc:1514] Processing DeleteTablet for tablet f362b4b0bc2248ea80dcaca8c6d627ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:23 UTC) from {username='slave'} at 127.0.0.1:49772
I20260810 07:25:23.221501 21210 tablet_service.cc:1514] Processing DeleteTablet for tablet f362b4b0bc2248ea80dcaca8c6d627ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:23 UTC) from {username='slave'} at 127.0.0.1:48606
I20260810 07:25:23.222220 21405 tablet_replica.cc:330] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64: stopping tablet replica
I20260810 07:25:23.222465 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 20625
I20260810 07:25:23.229547 21405 raft_consensus.cc:2229] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:25:23.230013 21405 raft_consensus.cc:2258] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:23.231586 21412 tablet_replica.cc:330] T f362b4b0bc2248ea80dcaca8c6d627ef P 303621182f8e4f47ae9a8c63445914d2: stopping tablet replica
I20260810 07:25:23.231905 21412 raft_consensus.cc:2229] T f362b4b0bc2248ea80dcaca8c6d627ef P 303621182f8e4f47ae9a8c63445914d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:25:23.232218 21412 raft_consensus.cc:2258] T f362b4b0bc2248ea80dcaca8c6d627ef P 303621182f8e4f47ae9a8c63445914d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:23.233580 21405 ts_tablet_manager.cc:1905] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:23.235424 21412 ts_tablet_manager.cc:1905] T f362b4b0bc2248ea80dcaca8c6d627ef P 303621182f8e4f47ae9a8c63445914d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:23.236155 21405 ts_tablet_manager.cc:1918] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:25:23.236267 21405 log.cc:1196] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-1/wal/wals/f362b4b0bc2248ea80dcaca8c6d627ef
I20260810 07:25:23.236632 21405 ts_tablet_manager.cc:1939] T f362b4b0bc2248ea80dcaca8c6d627ef P 5a7ebdfc6bc04b7395b454a41a7eec64: Deleting consensus metadata
I20260810 07:25:23.238052 20484 catalog_manager.cc:4841] TS 5a7ebdfc6bc04b7395b454a41a7eec64 (127.12.157.194:33719): tablet f362b4b0bc2248ea80dcaca8c6d627ef (table test-table [id=bf099b1d96e94ce9abd922e272068605]) successfully deleted
I20260810 07:25:23.238127 21412 ts_tablet_manager.cc:1918] T f362b4b0bc2248ea80dcaca8c6d627ef P 303621182f8e4f47ae9a8c63445914d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:25:23.238214 21412 log.cc:1196] T f362b4b0bc2248ea80dcaca8c6d627ef P 303621182f8e4f47ae9a8c63445914d2: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786346694975122-12919-0/minicluster-data/ts-2/wal/wals/f362b4b0bc2248ea80dcaca8c6d627ef
I20260810 07:25:23.238513 21412 ts_tablet_manager.cc:1939] T f362b4b0bc2248ea80dcaca8c6d627ef P 303621182f8e4f47ae9a8c63445914d2: Deleting consensus metadata
I20260810 07:25:23.239805 20484 catalog_manager.cc:4841] TS 303621182f8e4f47ae9a8c63445914d2 (127.12.157.195:46001): tablet f362b4b0bc2248ea80dcaca8c6d627ef (table test-table [id=bf099b1d96e94ce9abd922e272068605]) successfully deleted
W20260810 07:25:23.242959 20484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.157.193:37863 (error 108)
I20260810 07:25:23.243156 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 20856
I20260810 07:25:23.250093 21349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:23.249782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:37863 (local address 127.0.0.1:49788)
0810 07:25:23.249926 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:23.250011 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.193:37863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 07:25:23.250272 20484 catalog_manager.cc:4584] TS 3c7e1a58e302440d88d12b217956a9b5 (127.12.157.193:37863): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f362b4b0bc2248ea80dcaca8c6d627ef: Network error: Client connection negotiation failed: client connection to 127.12.157.193:37863: connect: Connection refused (error 111)
I20260810 07:25:23.255605 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 21089
I20260810 07:25:23.269433 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 20461
2026-08-10T07:25:23Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (993 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 07:25:23.299194 12919 mini_oidc.cc:152] Starting JWKS server
I20260810 07:25:23.300794 12919 webserver.cc:460] Webserver started at https://127.0.0.1:37097/ using document root <none> and password file <none>
I20260810 07:25:23.301067 12919 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 07:25:23.301348 12919 webserver.cc:460] Webserver started at http://127.0.0.1:38397/ using document root <none> and password file <none>
2026-08-10T07:25:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:25:23Z Disabled control of system clock
I20260810 07:25:23.311980 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:35267
--webserver_interface=127.12.157.254
--webserver_port=0
--builtin_ntp_servers=127.12.157.212:33091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:35267
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37097/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346694975122-12919-0/testchainca.cert with env {}
W20260810 07:25:23.423916 21426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:23.424190 21426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:23.424232 21426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:23.428313 21426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:25:23.428377 21426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:23.428393 21426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:23.428411 21426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:25:23.428426 21426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 07:25:23.428442 21426 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 07:25:23.428455 21426 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37097/jwks.json
I20260810 07:25:23.433892 21426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:33091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:35267
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:35267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37097/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346694975122-12919-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:23.434966 21426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:23.435926 21426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 07:25:23.444926 21426 server_base.cc:1034] running on GCE node
W20260810 07:25:23.444875 21434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:23.444819 21437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:23.444814 21435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:23.445765 21426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:23.446909 21426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:23.448092 21426 hybrid_clock.cc:648] HybridClock initialized: now 1786346723448067 us; error 37 us; skew 500 ppm
I20260810 07:25:23.450184 21426 webserver.cc:460] Webserver started at http://127.12.157.254:41385/ using document root <none> and password file <none>
I20260810 07:25:23.450781 21426 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:23.450848 21426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:23.451068 21426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:23.453431 21426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "d278fde3aab34030b01310585cb07fbe"
format_stamp: "Formatted at 2026-08-10 07:25:23 on dist-test-slave-ffrd"
server_key: "203ac18aa145e7806efd8853b446a026"
server_key_iv: "a3591c25a01e0c9250ece95968a178a4"
server_key_version: "encryptionkey@0"
I20260810 07:25:23.454105 21426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "d278fde3aab34030b01310585cb07fbe"
format_stamp: "Formatted at 2026-08-10 07:25:23 on dist-test-slave-ffrd"
server_key: "203ac18aa145e7806efd8853b446a026"
server_key_iv: "a3591c25a01e0c9250ece95968a178a4"
server_key_version: "encryptionkey@0"
I20260810 07:25:23.458003 21426 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 07:25:23.460493 21454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:23.461894 21426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 07:25:23.462051 21426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "d278fde3aab34030b01310585cb07fbe"
format_stamp: "Formatted at 2026-08-10 07:25:23 on dist-test-slave-ffrd"
server_key: "203ac18aa145e7806efd8853b446a026"
server_key_iv: "a3591c25a01e0c9250ece95968a178a4"
server_key_version: "encryptionkey@0"
I20260810 07:25:23.462170 21426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:23.498427 21426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:23.499192 21426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:23.499396 21426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:23.507165 21426 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:35267
I20260810 07:25:23.507175 21558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:35267 every 8 connection(s)
I20260810 07:25:23.508139 21426 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:25:23.511412 21560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:23.517010 21560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d278fde3aab34030b01310585cb07fbe: Bootstrap starting.
I20260810 07:25:23.518450 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 21426
I20260810 07:25:23.518592 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:25:23.518877 12919 external_mini_cluster.cc:1428] Setting key 0a10eba08b6fcdaa44d7a2799e6c8a0c
I20260810 07:25:23.519809 21560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d278fde3aab34030b01310585cb07fbe: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:23.520586 21560 log.cc:824] T 00000000000000000000000000000000 P d278fde3aab34030b01310585cb07fbe: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:23.522835 21560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d278fde3aab34030b01310585cb07fbe: No bootstrap required, opened a new log
I20260810 07:25:23.527082 21560 raft_consensus.cc:348] T 00000000000000000000000000000000 P d278fde3aab34030b01310585cb07fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d278fde3aab34030b01310585cb07fbe" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 35267 } }
I20260810 07:25:23.527374 21560 raft_consensus.cc:374] T 00000000000000000000000000000000 P d278fde3aab34030b01310585cb07fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:23.527446 21560 raft_consensus.cc:729] T 00000000000000000000000000000000 P d278fde3aab34030b01310585cb07fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d278fde3aab34030b01310585cb07fbe, State: Initialized, Role: FOLLOWER
I20260810 07:25:23.527858 21560 consensus_queue.cc:260] T 00000000000000000000000000000000 P d278fde3aab34030b01310585cb07fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d278fde3aab34030b01310585cb07fbe" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 35267 } }
I20260810 07:25:23.528000 21560 raft_consensus.cc:388] T 00000000000000000000000000000000 P d278fde3aab34030b01310585cb07fbe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:25:23.528081 21560 raft_consensus.cc:482] T 00000000000000000000000000000000 P d278fde3aab34030b01310585cb07fbe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:25:23.528195 21560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d278fde3aab34030b01310585cb07fbe [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:23.528410 21567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:23.519963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36834 (local address 127.12.157.254:35267)
0810 07:25:23.520654 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:23.520668 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:23.520721 (+ 53us) server_negotiation.cc:407] Connection header received
0810 07:25:23.521584 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:23.521604 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:23.521932 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:23.522200 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:23.522585 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:23.523614 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:23.524867 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:23.525381 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:23.525511 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:23.526431 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:23.526436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:23.526448 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:25:23.526820 (+ 372us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:23.526898 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:25:23.526918 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:23.526995 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:23.527163 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 07:25:23.527453 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":156,"threads_started":1}
I20260810 07:25:23.529381 21560 raft_consensus.cc:504] T 00000000000000000000000000000000 P d278fde3aab34030b01310585cb07fbe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d278fde3aab34030b01310585cb07fbe" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 35267 } }
I20260810 07:25:23.529721 21560 leader_election.cc:302] T 00000000000000000000000000000000 P d278fde3aab34030b01310585cb07fbe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d278fde3aab34030b01310585cb07fbe; no voters:
I20260810 07:25:23.530035 21560 leader_election.cc:288] T 00000000000000000000000000000000 P d278fde3aab34030b01310585cb07fbe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:25:23.530128 21573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d278fde3aab34030b01310585cb07fbe [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:23.530383 21573 raft_consensus.cc:686] T 00000000000000000000000000000000 P d278fde3aab34030b01310585cb07fbe [term 1 LEADER]: Becoming Leader. State: Replica: d278fde3aab34030b01310585cb07fbe, State: Running, Role: LEADER
I20260810 07:25:23.530694 21573 consensus_queue.cc:237] T 00000000000000000000000000000000 P d278fde3aab34030b01310585cb07fbe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d278fde3aab34030b01310585cb07fbe" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 35267 } }
I20260810 07:25:23.531352 21560 sys_catalog.cc:564] T 00000000000000000000000000000000 P d278fde3aab34030b01310585cb07fbe [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:25:23.531656 12919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:25:23.532338 21583 sys_catalog.cc:455] T 00000000000000000000000000000000 P d278fde3aab34030b01310585cb07fbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d278fde3aab34030b01310585cb07fbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d278fde3aab34030b01310585cb07fbe" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 35267 } } }
I20260810 07:25:23.532464 21583 sys_catalog.cc:458] T 00000000000000000000000000000000 P d278fde3aab34030b01310585cb07fbe [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:23.532627 21575 sys_catalog.cc:455] T 00000000000000000000000000000000 P d278fde3aab34030b01310585cb07fbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader d278fde3aab34030b01310585cb07fbe. Latest consensus state: current_term: 1 leader_uuid: "d278fde3aab34030b01310585cb07fbe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d278fde3aab34030b01310585cb07fbe" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 35267 } } }
I20260810 07:25:23.533022 21575 sys_catalog.cc:458] T 00000000000000000000000000000000 P d278fde3aab34030b01310585cb07fbe [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:23.534754 21591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:25:23.538034 21591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:25:23.544127 21591 catalog_manager.cc:1347] Generated new cluster ID: cfbbb877d13e423f95ae3b5044fb2d25
I20260810 07:25:23.544224 21591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:25:23.555611 21591 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:25:23.556615 21591 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:25:23.565464 21591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d278fde3aab34030b01310585cb07fbe: Generated new TSK 0
I20260810 07:25:23.566258 21591 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 07:25:23.606765 21626 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 07:25:23.606953 21626 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 07:25:23.607439 21567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:23.582098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36844 (local address 127.12.157.254:35267)
0810 07:25:23.582244 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:23.582249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:23.582436 (+ 187us) server_negotiation.cc:407] Connection header received
0810 07:25:23.582671 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:23.582674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:23.582726 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:23.582816 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0810 07:25:23.583132 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:23.583694 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:23.585041 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:23.585293 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:23.585394 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:23.585998 (+ 604us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 07:25:23.606947 (+ 20949us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 07:25:23.607301 (+ 354us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36844: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":59}
W20260810 07:25:23.607549 21567 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36844: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 07:25:23.608879 21621 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:23.581931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:35267 (local address 127.0.0.1:36844)
0810 07:25:23.582337 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:23.582354 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:23.582503 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:23.582857 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:23.582860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:23.582869 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0810 07:25:23.583015 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:23.583022 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:23.583806 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:23.583809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:23.584828 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:23.584841 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:23.585542 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:23.585547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:23.585664 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:23.585737 (+ 73us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 07:25:23.607255 (+ 21518us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 07:25:23.607524 (+ 269us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.157.254:35267: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":130,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.12.157.254:35267: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37097/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 07:25:23.609835 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 21426
2026-08-10T07:25:23Z chronyd exiting
I20260810 07:25:23.658214 12919 test_util.cc:175] -----------------------------------------------
I20260810 07:25:23.658330 12919 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786346694975122-12919-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (360 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 07:25:23.659456 12919 mini_oidc.cc:152] Starting JWKS server
I20260810 07:25:23.660338 12919 webserver.cc:460] Webserver started at https://127.0.0.1:36877/ using document root <none> and password file <none>
I20260810 07:25:23.660467 12919 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 07:25:23.660738 12919 webserver.cc:460] Webserver started at http://127.0.0.1:37123/ using document root <none> and password file <none>
2026-08-10T07:25:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:25:23Z Disabled control of system clock
I20260810 07:25:23.671438 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:39233
--webserver_interface=127.12.157.254
--webserver_port=0
--builtin_ntp_servers=127.12.157.212:33033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:39233
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36877/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346694975122-12919-0/testchainca.cert with env {}
W20260810 07:25:23.788798 21633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:23.789119 21633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:23.789201 21633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:23.793251 21633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:25:23.793309 21633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:23.793325 21633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:23.793342 21633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:25:23.793357 21633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 07:25:23.793419 21633 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 07:25:23.793447 21633 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36877/jwks.json
I20260810 07:25:23.798897 21633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:33033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:39233
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:39233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36877/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346694975122-12919-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:23.800005 21633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:23.801060 21633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:23.809960 21644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:23.809986 21641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:23.809985 21642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:23.810289 21633 server_base.cc:1034] running on GCE node
I20260810 07:25:23.810918 21633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:23.812027 21633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:23.813190 21633 hybrid_clock.cc:648] HybridClock initialized: now 1786346723813165 us; error 36 us; skew 500 ppm
I20260810 07:25:23.815179 21633 webserver.cc:460] Webserver started at http://127.12.157.254:41891/ using document root <none> and password file <none>
I20260810 07:25:23.815781 21633 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:23.815851 21633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:23.816084 21633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:23.818485 21633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "2ba168f4fdb74a5094a1c25a3b156415"
format_stamp: "Formatted at 2026-08-10 07:25:23 on dist-test-slave-ffrd"
server_key: "46f31df1fcab6c4994b6535b283d4532"
server_key_iv: "9eb64c122a65eef35d181202786a3ab5"
server_key_version: "encryptionkey@0"
I20260810 07:25:23.819151 21633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "2ba168f4fdb74a5094a1c25a3b156415"
format_stamp: "Formatted at 2026-08-10 07:25:23 on dist-test-slave-ffrd"
server_key: "46f31df1fcab6c4994b6535b283d4532"
server_key_iv: "9eb64c122a65eef35d181202786a3ab5"
server_key_version: "encryptionkey@0"
I20260810 07:25:23.822924 21633 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 07:25:23.825398 21657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:23.826668 21633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:25:23.826812 21633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "2ba168f4fdb74a5094a1c25a3b156415"
format_stamp: "Formatted at 2026-08-10 07:25:23 on dist-test-slave-ffrd"
server_key: "46f31df1fcab6c4994b6535b283d4532"
server_key_iv: "9eb64c122a65eef35d181202786a3ab5"
server_key_version: "encryptionkey@0"
I20260810 07:25:23.826917 21633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:23.840565 21633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:23.841373 21633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:23.841580 21633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:23.848876 21633 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:39233
I20260810 07:25:23.848929 21742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:39233 every 8 connection(s)
I20260810 07:25:23.849954 21633 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:25:23.853377 21746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:23.858045 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 21633
I20260810 07:25:23.858206 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:25:23.858522 12919 external_mini_cluster.cc:1428] Setting key 6cd937dbd6814663be9c797102176f18
I20260810 07:25:23.860284 21746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ba168f4fdb74a5094a1c25a3b156415: Bootstrap starting.
I20260810 07:25:23.862744 21746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ba168f4fdb74a5094a1c25a3b156415: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:23.863502 21746 log.cc:824] T 00000000000000000000000000000000 P 2ba168f4fdb74a5094a1c25a3b156415: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:23.865841 21746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ba168f4fdb74a5094a1c25a3b156415: No bootstrap required, opened a new log
I20260810 07:25:23.869024 21750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:23.860059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54170 (local address 127.12.157.254:39233)
0810 07:25:23.860892 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:23.860912 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:23.860997 (+ 85us) server_negotiation.cc:407] Connection header received
0810 07:25:23.861694 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:23.861740 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:23.862170 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:23.862517 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:23.862993 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:23.864159 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:23.865501 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:23.866066 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:23.866197 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:23.867120 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:23.867125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:23.867138 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:25:23.867554 (+ 416us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:23.867614 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:25:23.867634 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:23.867714 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:23.867896 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 07:25:23.868201 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":170,"threads_started":1}
I20260810 07:25:23.870013 21746 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ba168f4fdb74a5094a1c25a3b156415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba168f4fdb74a5094a1c25a3b156415" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39233 } }
I20260810 07:25:23.870301 21746 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ba168f4fdb74a5094a1c25a3b156415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:23.870361 21746 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ba168f4fdb74a5094a1c25a3b156415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ba168f4fdb74a5094a1c25a3b156415, State: Initialized, Role: FOLLOWER
I20260810 07:25:23.870831 21746 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ba168f4fdb74a5094a1c25a3b156415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba168f4fdb74a5094a1c25a3b156415" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39233 } }
I20260810 07:25:23.871006 21746 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ba168f4fdb74a5094a1c25a3b156415 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:25:23.871081 21746 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ba168f4fdb74a5094a1c25a3b156415 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:25:23.871191 21746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ba168f4fdb74a5094a1c25a3b156415 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:23.872102 12919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:25:23.872464 21746 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ba168f4fdb74a5094a1c25a3b156415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba168f4fdb74a5094a1c25a3b156415" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39233 } }
I20260810 07:25:23.872819 21746 leader_election.cc:302] T 00000000000000000000000000000000 P 2ba168f4fdb74a5094a1c25a3b156415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ba168f4fdb74a5094a1c25a3b156415; no voters:
I20260810 07:25:23.873184 21746 leader_election.cc:288] T 00000000000000000000000000000000 P 2ba168f4fdb74a5094a1c25a3b156415 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:25:23.873265 21754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ba168f4fdb74a5094a1c25a3b156415 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:23.873939 21754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ba168f4fdb74a5094a1c25a3b156415 [term 1 LEADER]: Becoming Leader. State: Replica: 2ba168f4fdb74a5094a1c25a3b156415, State: Running, Role: LEADER
I20260810 07:25:23.874253 21754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ba168f4fdb74a5094a1c25a3b156415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba168f4fdb74a5094a1c25a3b156415" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39233 } }
I20260810 07:25:23.874543 21746 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ba168f4fdb74a5094a1c25a3b156415 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:25:23.876087 21757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ba168f4fdb74a5094a1c25a3b156415 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ba168f4fdb74a5094a1c25a3b156415. Latest consensus state: current_term: 1 leader_uuid: "2ba168f4fdb74a5094a1c25a3b156415" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba168f4fdb74a5094a1c25a3b156415" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39233 } } }
I20260810 07:25:23.876120 21756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ba168f4fdb74a5094a1c25a3b156415 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ba168f4fdb74a5094a1c25a3b156415" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ba168f4fdb74a5094a1c25a3b156415" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39233 } } }
I20260810 07:25:23.876255 21757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ba168f4fdb74a5094a1c25a3b156415 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:23.876259 21756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ba168f4fdb74a5094a1c25a3b156415 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:23.876660 21790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:25:23.879470 21790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:25:23.885190 21790 catalog_manager.cc:1347] Generated new cluster ID: 91db56d0e5614ee3925d00125a8ed9dd
I20260810 07:25:23.885267 21790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:25:23.902376 21790 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:25:23.903656 21790 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:25:23.912717 21790 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ba168f4fdb74a5094a1c25a3b156415: Generated new TSK 0
I20260810 07:25:23.913719 21790 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 07:25:23.951252 21817 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 07:25:23.951393 21817 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 07:25:23.952158 21750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:23.925532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54178 (local address 127.12.157.254:39233)
0810 07:25:23.925679 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:23.925684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:23.929363 (+ 3679us) server_negotiation.cc:407] Connection header received
0810 07:25:23.929632 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:23.929635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:23.929697 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:23.929797 (+ 100us) server_negotiation.cc:226] Negotiated authn=JWT
0810 07:25:23.930116 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:23.930808 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:23.932154 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:23.932413 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:23.932500 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:23.932862 (+ 362us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 07:25:23.951683 (+ 18821us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36877/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 07:25:23.952005 (+ 322us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54178: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36877/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 07:25:23.952234 21750 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54178: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36877/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 07:25:23.952289 21815 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:23.925433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39233 (local address 127.0.0.1:54178)
0810 07:25:23.929222 (+ 3789us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:23.929241 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:23.929457 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:23.929819 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:23.929823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:23.929833 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0810 07:25:23.929997 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:23.930004 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:23.930925 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:23.930928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:23.931988 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:23.932001 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:23.932532 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:23.932536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:23.932631 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:23.932665 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 07:25:23.951916 (+ 19251us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36877/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 07:25:23.952102 (+ 186us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.157.254:39233: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36877/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":3676,"thread_start_us":127,"threads_started":1}
I20260810 07:25:23.953132 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 21633
2026-08-10T07:25:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (343 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 07:25:24.002810 12919 mini_oidc.cc:152] Starting JWKS server
I20260810 07:25:24.003700 12919 webserver.cc:460] Webserver started at https://127.0.0.1:37385/ using document root <none> and password file <none>
I20260810 07:25:24.003788 12919 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 07:25:24.004051 12919 webserver.cc:460] Webserver started at http://127.0.0.1:35563/ using document root <none> and password file <none>
2026-08-10T07:25:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:25:24Z Disabled control of system clock
I20260810 07:25:24.015388 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:45393
--webserver_interface=127.12.157.254
--webserver_port=0
--builtin_ntp_servers=127.12.157.212:45569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:45393
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37385/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 07:25:24.129724 21826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:24.130115 21826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:24.130192 21826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:24.134824 21826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:25:24.134935 21826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:24.134964 21826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:24.135013 21826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:25:24.135053 21826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 07:25:24.135093 21826 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 07:25:24.135139 21826 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37385/jwks.json
I20260810 07:25:24.142180 21826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:45569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:45393
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:45393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37385/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:24.143471 21826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:24.145027 21826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:24.154183 21835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:24.154191 21831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:24.154198 21833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:24.154683 21826 server_base.cc:1034] running on GCE node
I20260810 07:25:24.155290 21826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:24.156796 21826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:24.158003 21826 hybrid_clock.cc:648] HybridClock initialized: now 1786346724157972 us; error 44 us; skew 500 ppm
I20260810 07:25:24.160033 21826 webserver.cc:460] Webserver started at http://127.12.157.254:36811/ using document root <none> and password file <none>
I20260810 07:25:24.160617 21826 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:24.160682 21826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:24.160903 21826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:24.163028 21826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "ac083184b1c140ab844aa5b5ebd38168"
format_stamp: "Formatted at 2026-08-10 07:25:24 on dist-test-slave-ffrd"
server_key: "7c68ed79d1cf336e218a2db7cf6101a3"
server_key_iv: "01d37e48424774a8a593c10d3bf891ce"
server_key_version: "encryptionkey@0"
I20260810 07:25:24.163635 21826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "ac083184b1c140ab844aa5b5ebd38168"
format_stamp: "Formatted at 2026-08-10 07:25:24 on dist-test-slave-ffrd"
server_key: "7c68ed79d1cf336e218a2db7cf6101a3"
server_key_iv: "01d37e48424774a8a593c10d3bf891ce"
server_key_version: "encryptionkey@0"
I20260810 07:25:24.167874 21826 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260810 07:25:24.170533 21847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:24.171856 21826 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 07:25:24.172008 21826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "ac083184b1c140ab844aa5b5ebd38168"
format_stamp: "Formatted at 2026-08-10 07:25:24 on dist-test-slave-ffrd"
server_key: "7c68ed79d1cf336e218a2db7cf6101a3"
server_key_iv: "01d37e48424774a8a593c10d3bf891ce"
server_key_version: "encryptionkey@0"
I20260810 07:25:24.172132 21826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:24.206355 21826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:24.207140 21826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:24.207321 21826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:24.214789 21826 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:45393
I20260810 07:25:24.214829 21937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:45393 every 8 connection(s)
I20260810 07:25:24.215758 21826 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:25:24.218676 21939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:24.222417 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 21826
I20260810 07:25:24.222543 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:25:24.222815 12919 external_mini_cluster.cc:1428] Setting key 5642c753fbe519440ba0079de54b2b89
I20260810 07:25:24.225047 21939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac083184b1c140ab844aa5b5ebd38168: Bootstrap starting.
I20260810 07:25:24.227257 21939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac083184b1c140ab844aa5b5ebd38168: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:24.228012 21939 log.cc:824] T 00000000000000000000000000000000 P ac083184b1c140ab844aa5b5ebd38168: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:24.230019 21939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac083184b1c140ab844aa5b5ebd38168: No bootstrap required, opened a new log
I20260810 07:25:24.232717 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:24.224305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56850 (local address 127.12.157.254:45393)
0810 07:25:24.225043 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:24.225060 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:24.225098 (+ 38us) server_negotiation.cc:407] Connection header received
0810 07:25:24.225768 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:24.225786 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:24.226165 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:24.226509 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:24.226962 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:24.228009 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:24.228916 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:24.229441 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:24.229611 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:24.230595 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:24.230601 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:24.230614 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:25:24.230998 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:24.231055 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:25:24.231075 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:24.231214 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:24.231337 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 07:25:24.231600 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":145,"threads_started":1}
I20260810 07:25:24.233315 21939 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac083184b1c140ab844aa5b5ebd38168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac083184b1c140ab844aa5b5ebd38168" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 45393 } }
I20260810 07:25:24.233536 21939 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac083184b1c140ab844aa5b5ebd38168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:24.233603 21939 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac083184b1c140ab844aa5b5ebd38168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac083184b1c140ab844aa5b5ebd38168, State: Initialized, Role: FOLLOWER
I20260810 07:25:24.234112 21939 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac083184b1c140ab844aa5b5ebd38168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac083184b1c140ab844aa5b5ebd38168" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 45393 } }
I20260810 07:25:24.234275 21939 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac083184b1c140ab844aa5b5ebd38168 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:25:24.234342 21939 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac083184b1c140ab844aa5b5ebd38168 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:25:24.234442 21939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac083184b1c140ab844aa5b5ebd38168 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:24.235627 21939 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac083184b1c140ab844aa5b5ebd38168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac083184b1c140ab844aa5b5ebd38168" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 45393 } }
I20260810 07:25:24.235817 12919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:25:24.236009 21939 leader_election.cc:302] T 00000000000000000000000000000000 P ac083184b1c140ab844aa5b5ebd38168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac083184b1c140ab844aa5b5ebd38168; no voters:
I20260810 07:25:24.236374 21939 leader_election.cc:288] T 00000000000000000000000000000000 P ac083184b1c140ab844aa5b5ebd38168 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:25:24.236526 21949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac083184b1c140ab844aa5b5ebd38168 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:24.236769 21949 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac083184b1c140ab844aa5b5ebd38168 [term 1 LEADER]: Becoming Leader. State: Replica: ac083184b1c140ab844aa5b5ebd38168, State: Running, Role: LEADER
I20260810 07:25:24.237156 21949 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac083184b1c140ab844aa5b5ebd38168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac083184b1c140ab844aa5b5ebd38168" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 45393 } }
I20260810 07:25:24.237741 21939 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac083184b1c140ab844aa5b5ebd38168 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:25:24.238958 21955 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac083184b1c140ab844aa5b5ebd38168 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac083184b1c140ab844aa5b5ebd38168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac083184b1c140ab844aa5b5ebd38168" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 45393 } } }
I20260810 07:25:24.239106 21955 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac083184b1c140ab844aa5b5ebd38168 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:24.239400 21950 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac083184b1c140ab844aa5b5ebd38168 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac083184b1c140ab844aa5b5ebd38168. Latest consensus state: current_term: 1 leader_uuid: "ac083184b1c140ab844aa5b5ebd38168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac083184b1c140ab844aa5b5ebd38168" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 45393 } } }
I20260810 07:25:24.239506 21950 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac083184b1c140ab844aa5b5ebd38168 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:24.239620 21969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:25:24.243259 21969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:25:24.249099 21969 catalog_manager.cc:1347] Generated new cluster ID: b688e1eef8f0469bae1bbd7ae0edc121
I20260810 07:25:24.249178 21969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:25:24.268793 21969 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:25:24.269759 21969 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:25:24.277191 21969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac083184b1c140ab844aa5b5ebd38168: Generated new TSK 0
I20260810 07:25:24.277912 21969 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:25:24.295356 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:24.290211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56852 (local address 127.12.157.254:45393)
0810 07:25:24.290343 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:24.290347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:24.290572 (+ 225us) server_negotiation.cc:407] Connection header received
0810 07:25:24.290754 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:24.290756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:24.290807 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:24.290918 (+ 111us) server_negotiation.cc:226] Negotiated authn=JWT
0810 07:25:24.291181 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:24.291711 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:24.293011 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:24.293242 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:24.293321 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:24.293675 (+ 354us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 07:25:24.294962 (+ 1287us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37385/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 07:25:24.295221 (+ 259us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56852: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37385/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":51}
W20260810 07:25:24.295454 21943 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56852: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37385/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 07:25:24.298332 21996 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:24.290161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:45393 (local address 127.0.0.1:56852)
0810 07:25:24.290469 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:24.290485 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:24.290624 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:24.290921 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:24.290924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:24.290932 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0810 07:25:24.291068 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:24.291074 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:24.291819 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:24.291824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:24.292845 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:24.292855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:24.293367 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:24.293371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:24.293475 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:24.293506 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 07:25:24.295141 (+ 1635us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37385/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 07:25:24.298159 (+ 3018us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.157.254:45393: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37385/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":2788,"thread_start_us":118,"threads_started":1}
I20260810 07:25:24.299134 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 21826
E20260810 07:25:24.321540 21997 webserver.cc:533] Webserver: sslize failed:
E20260810 07:25:24.321661 21997 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-10T07:25:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (349 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T07:25:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:25:24Z Disabled control of system clock
I20260810 07:25:24.362473 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:44469
--webserver_interface=127.12.157.254
--webserver_port=0
--builtin_ntp_servers=127.12.157.212:39487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:44469
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346694975122-12919-0/testchainca.cert with env {}
W20260810 07:25:24.475111 22002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:24.475373 22002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:24.475420 22002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:24.479405 22002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:25:24.479467 22002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:24.479483 22002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:24.479501 22002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:25:24.479565 22002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 07:25:24.479598 22002 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 07:25:24.479630 22002 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 07:25:24.485656 22002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:39487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:44469
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:44469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346694975122-12919-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:24.486773 22002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:24.487838 22002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:24.496630 22014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:24.496636 22017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:24.496656 22013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:24.497004 22002 server_base.cc:1034] running on GCE node
I20260810 07:25:24.497546 22002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:24.498587 22002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:24.499754 22002 hybrid_clock.cc:648] HybridClock initialized: now 1786346724499738 us; error 34 us; skew 500 ppm
I20260810 07:25:24.501694 22002 webserver.cc:460] Webserver started at http://127.12.157.254:37585/ using document root <none> and password file <none>
I20260810 07:25:24.502218 22002 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:24.502275 22002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:24.502444 22002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:24.504408 22002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "d9f602e147b24a40a288cb96ca1d9f20"
format_stamp: "Formatted at 2026-08-10 07:25:24 on dist-test-slave-ffrd"
server_key: "90ffec254e1e1368c3ca2fa7f286d0b0"
server_key_iv: "59f64c7c9549a59e8929109dd26106fc"
server_key_version: "encryptionkey@0"
I20260810 07:25:24.504935 22002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "d9f602e147b24a40a288cb96ca1d9f20"
format_stamp: "Formatted at 2026-08-10 07:25:24 on dist-test-slave-ffrd"
server_key: "90ffec254e1e1368c3ca2fa7f286d0b0"
server_key_iv: "59f64c7c9549a59e8929109dd26106fc"
server_key_version: "encryptionkey@0"
I20260810 07:25:24.508565 22002 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 07:25:24.510944 22025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:24.512142 22002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 07:25:24.512243 22002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "d9f602e147b24a40a288cb96ca1d9f20"
format_stamp: "Formatted at 2026-08-10 07:25:24 on dist-test-slave-ffrd"
server_key: "90ffec254e1e1368c3ca2fa7f286d0b0"
server_key_iv: "59f64c7c9549a59e8929109dd26106fc"
server_key_version: "encryptionkey@0"
I20260810 07:25:24.512346 22002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:24.528034 22002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:24.528749 22002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:24.528915 22002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:24.536278 22002 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:44469
I20260810 07:25:24.536284 22113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:44469 every 8 connection(s)
I20260810 07:25:24.537297 22002 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:25:24.538661 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 22002
I20260810 07:25:24.538763 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:25:24.539063 12919 external_mini_cluster.cc:1428] Setting key bad5c60f64343942e9e0058dd8acfa9a
I20260810 07:25:24.540755 22114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:24.546516 22114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9f602e147b24a40a288cb96ca1d9f20: Bootstrap starting.
I20260810 07:25:24.549014 22120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:24.540450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50242 (local address 127.12.157.254:44469)
0810 07:25:24.541233 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:24.541257 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:24.541308 (+ 51us) server_negotiation.cc:407] Connection header received
0810 07:25:24.542013 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:24.542031 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:24.542437 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:24.542785 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:24.543226 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:24.544203 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:24.545188 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:24.545653 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:24.545833 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:24.546871 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:24.546877 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:24.546892 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:25:24.547389 (+ 497us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:24.547463 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:25:24.547491 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:24.547598 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:24.547756 (+ 158us) server_negotiation.cc:299] Negotiation successful
0810 07:25:24.548058 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":149,"threads_started":1}
I20260810 07:25:24.549386 22114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9f602e147b24a40a288cb96ca1d9f20: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:24.550127 22114 log.cc:824] T 00000000000000000000000000000000 P d9f602e147b24a40a288cb96ca1d9f20: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:24.552024 12919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:25:24.552186 22114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9f602e147b24a40a288cb96ca1d9f20: No bootstrap required, opened a new log
I20260810 07:25:24.555907 22114 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9f602e147b24a40a288cb96ca1d9f20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f602e147b24a40a288cb96ca1d9f20" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44469 } }
I20260810 07:25:24.556133 22114 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9f602e147b24a40a288cb96ca1d9f20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:24.556178 22114 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9f602e147b24a40a288cb96ca1d9f20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9f602e147b24a40a288cb96ca1d9f20, State: Initialized, Role: FOLLOWER
I20260810 07:25:24.556537 22114 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9f602e147b24a40a288cb96ca1d9f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f602e147b24a40a288cb96ca1d9f20" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44469 } }
I20260810 07:25:24.556648 22114 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9f602e147b24a40a288cb96ca1d9f20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:25:24.556685 22114 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9f602e147b24a40a288cb96ca1d9f20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:25:24.556756 22114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9f602e147b24a40a288cb96ca1d9f20 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:24.557758 22114 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9f602e147b24a40a288cb96ca1d9f20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f602e147b24a40a288cb96ca1d9f20" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44469 } }
I20260810 07:25:24.558068 22114 leader_election.cc:302] T 00000000000000000000000000000000 P d9f602e147b24a40a288cb96ca1d9f20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9f602e147b24a40a288cb96ca1d9f20; no voters:
I20260810 07:25:24.558332 22114 leader_election.cc:288] T 00000000000000000000000000000000 P d9f602e147b24a40a288cb96ca1d9f20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:25:24.558482 22123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9f602e147b24a40a288cb96ca1d9f20 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:24.558738 22123 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9f602e147b24a40a288cb96ca1d9f20 [term 1 LEADER]: Becoming Leader. State: Replica: d9f602e147b24a40a288cb96ca1d9f20, State: Running, Role: LEADER
I20260810 07:25:24.559058 22123 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9f602e147b24a40a288cb96ca1d9f20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f602e147b24a40a288cb96ca1d9f20" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44469 } }
I20260810 07:25:24.559579 22114 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9f602e147b24a40a288cb96ca1d9f20 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:25:24.560811 22124 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f602e147b24a40a288cb96ca1d9f20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9f602e147b24a40a288cb96ca1d9f20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f602e147b24a40a288cb96ca1d9f20" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44469 } } }
I20260810 07:25:24.560916 22124 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f602e147b24a40a288cb96ca1d9f20 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:24.561390 22135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:25:24.561347 22125 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9f602e147b24a40a288cb96ca1d9f20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9f602e147b24a40a288cb96ca1d9f20. Latest consensus state: current_term: 1 leader_uuid: "d9f602e147b24a40a288cb96ca1d9f20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9f602e147b24a40a288cb96ca1d9f20" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44469 } } }
I20260810 07:25:24.561444 22125 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9f602e147b24a40a288cb96ca1d9f20 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:24.564554 22135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:25:24.570453 22135 catalog_manager.cc:1347] Generated new cluster ID: 8089d4b33f2244b9aa285faba8c9d53e
I20260810 07:25:24.570531 22135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:25:24.590493 22135 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:25:24.591871 22135 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:25:24.604705 22135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9f602e147b24a40a288cb96ca1d9f20: Generated new TSK 0
I20260810 07:25:24.605636 22135 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:25:24.689237 22120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:24.615536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50246 (local address 127.12.157.254:44469)
0810 07:25:24.615673 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:24.615678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:24.615810 (+ 132us) server_negotiation.cc:407] Connection header received
0810 07:25:24.615992 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:24.615995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:24.616078 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:24.616205 (+ 127us) server_negotiation.cc:226] Negotiated authn=JWT
0810 07:25:24.616484 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:24.617179 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:24.618047 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:24.618291 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:24.618396 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:24.618737 (+ 341us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 07:25:24.688635 (+ 69898us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 07:25:24.689066 (+ 431us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50246: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":44}
W20260810 07:25:24.689339 22120 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50246: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 07:25:24.689644 22167 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:24.615359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:44469 (local address 127.0.0.1:50246)
0810 07:25:24.615702 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:24.615717 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:24.615851 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:24.616204 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:24.616208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:24.616217 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0810 07:25:24.616365 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:24.616371 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:24.617291 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:24.617295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:24.617923 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:24.617930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:24.618429 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:24.618435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:24.618530 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:24.618553 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 07:25:24.688992 (+ 70439us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 07:25:24.689204 (+ 212us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.157.254:44469: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":125,"threads_started":1}
I20260810 07:25:24.690460 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 22002
2026-08-10T07:25:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (370 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (137 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (131 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:25:25 dist-test-slave-ffrd krb5kdc[22189](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:25:25 dist-test-slave-ffrd krb5kdc[22189](info): set up 2 sockets
Aug 10 07:25:25 dist-test-slave-ffrd krb5kdc[22189](info): commencing operation
krb5kdc: starting...
W20260810 07:25:26.238644 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.247s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:25:26 dist-test-slave-ffrd krb5kdc[22189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:25:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:25:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:25:26.372815 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:37821
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:38007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:37821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:26.494251 22208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:26.494541 22208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:26.494625 22208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:26.498591 22208 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:25:26.498670 22208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:26.498706 22208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:26.498767 22208 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:25:26.498821 22208 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:25:26.504086 22208 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:38007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:37821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:37821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:26.505295 22208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:26.506321 22208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:26.515235 22215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:26.515225 22214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:26.515239 22217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:26.515399 22208 server_base.cc:1034] running on GCE node
I20260810 07:25:26.516109 22208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:26.517328 22208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:26.518509 22208 hybrid_clock.cc:648] HybridClock initialized: now 1786346726518479 us; error 43 us; skew 500 ppm
Aug 10 07:25:26 dist-test-slave-ffrd krb5kdc[22189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346726, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:26.525667 22208 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:25:26.526710 22208 webserver.cc:460] Webserver started at http://127.12.157.254:39205/ using document root <none> and password file <none>
I20260810 07:25:26.527355 22208 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:26.527428 22208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:26.527663 22208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:26.530014 22208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "f925d30573ec4495b3d3229127b1ee9b"
format_stamp: "Formatted at 2026-08-10 07:25:26 on dist-test-slave-ffrd"
server_key: "250da60e4c50d8ac25b7c684ee4aa43e"
server_key_iv: "75cd78d18b39a8e19a1346dd80b073f5"
server_key_version: "encryptionkey@0"
I20260810 07:25:26.530637 22208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "f925d30573ec4495b3d3229127b1ee9b"
format_stamp: "Formatted at 2026-08-10 07:25:26 on dist-test-slave-ffrd"
server_key: "250da60e4c50d8ac25b7c684ee4aa43e"
server_key_iv: "75cd78d18b39a8e19a1346dd80b073f5"
server_key_version: "encryptionkey@0"
I20260810 07:25:26.534715 22208 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 07:25:26.537590 22233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:26.538928 22208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:25:26.539078 22208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "f925d30573ec4495b3d3229127b1ee9b"
format_stamp: "Formatted at 2026-08-10 07:25:26 on dist-test-slave-ffrd"
server_key: "250da60e4c50d8ac25b7c684ee4aa43e"
server_key_iv: "75cd78d18b39a8e19a1346dd80b073f5"
server_key_version: "encryptionkey@0"
I20260810 07:25:26.539194 22208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:26.572441 22208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:26.576515 22208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:26.576735 22208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:26.589759 22319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:37821 every 8 connection(s)
I20260810 07:25:26.589781 22208 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:37821
I20260810 07:25:26.591360 22208 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:25:26.593849 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 22208
I20260810 07:25:26.593997 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:25:26.594391 12919 external_mini_cluster.cc:1428] Setting key 0f278c24667af2860f9decaec4608e14
I20260810 07:25:26.594583 22320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:26.603859 22320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f925d30573ec4495b3d3229127b1ee9b: Bootstrap starting.
I20260810 07:25:26.607316 22320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f925d30573ec4495b3d3229127b1ee9b: Neither blocks nor log segments found. Creating new log.
Aug 10 07:25:26 dist-test-slave-ffrd krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:26.608353 22320 log.cc:824] T 00000000000000000000000000000000 P f925d30573ec4495b3d3229127b1ee9b: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:26.611131 22320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f925d30573ec4495b3d3229127b1ee9b: No bootstrap required, opened a new log
I20260810 07:25:26.614295 22323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:26.596163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40420 (local address 127.12.157.254:37821)
0810 07:25:26.597204 (+ 1041us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:26.597247 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:26.597309 (+ 62us) server_negotiation.cc:407] Connection header received
0810 07:25:26.598457 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:26.598485 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:26.598926 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:26.599317 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:26.600143 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:26.601233 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:26.602281 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:26.602791 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:26.603446 (+ 655us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:26.608398 (+ 4952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:26.608406 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:26.608424 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:26.608462 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:26.610583 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:26.611162 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:26.611169 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:26.611175 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:26.611356 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:26.611668 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:26.611672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:26.611673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:26.612083 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:26.612285 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:26.612977 (+ 692us) server_negotiation.cc:299] Negotiation successful
0810 07:25:26.613305 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":830,"thread_start_us":640,"threads_started":1}
I20260810 07:25:26.617445 22320 raft_consensus.cc:348] T 00000000000000000000000000000000 P f925d30573ec4495b3d3229127b1ee9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f925d30573ec4495b3d3229127b1ee9b" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37821 } }
I20260810 07:25:26.617782 22320 raft_consensus.cc:374] T 00000000000000000000000000000000 P f925d30573ec4495b3d3229127b1ee9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:26.617858 22320 raft_consensus.cc:729] T 00000000000000000000000000000000 P f925d30573ec4495b3d3229127b1ee9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f925d30573ec4495b3d3229127b1ee9b, State: Initialized, Role: FOLLOWER
I20260810 07:25:26.618405 22320 consensus_queue.cc:260] T 00000000000000000000000000000000 P f925d30573ec4495b3d3229127b1ee9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f925d30573ec4495b3d3229127b1ee9b" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37821 } }
I20260810 07:25:26.618595 22320 raft_consensus.cc:388] T 00000000000000000000000000000000 P f925d30573ec4495b3d3229127b1ee9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:25:26.618680 22320 raft_consensus.cc:482] T 00000000000000000000000000000000 P f925d30573ec4495b3d3229127b1ee9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:25:26.618814 22320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f925d30573ec4495b3d3229127b1ee9b [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:26.620348 22320 raft_consensus.cc:504] T 00000000000000000000000000000000 P f925d30573ec4495b3d3229127b1ee9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f925d30573ec4495b3d3229127b1ee9b" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37821 } }
I20260810 07:25:26.620823 22320 leader_election.cc:302] T 00000000000000000000000000000000 P f925d30573ec4495b3d3229127b1ee9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f925d30573ec4495b3d3229127b1ee9b; no voters:
I20260810 07:25:26.621387 22320 leader_election.cc:288] T 00000000000000000000000000000000 P f925d30573ec4495b3d3229127b1ee9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:25:26.621492 22325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f925d30573ec4495b3d3229127b1ee9b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:26.621752 22325 raft_consensus.cc:686] T 00000000000000000000000000000000 P f925d30573ec4495b3d3229127b1ee9b [term 1 LEADER]: Becoming Leader. State: Replica: f925d30573ec4495b3d3229127b1ee9b, State: Running, Role: LEADER
I20260810 07:25:26.622149 22325 consensus_queue.cc:237] T 00000000000000000000000000000000 P f925d30573ec4495b3d3229127b1ee9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f925d30573ec4495b3d3229127b1ee9b" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37821 } }
I20260810 07:25:26.622963 22320 sys_catalog.cc:564] T 00000000000000000000000000000000 P f925d30573ec4495b3d3229127b1ee9b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:25:26.624048 22327 sys_catalog.cc:455] T 00000000000000000000000000000000 P f925d30573ec4495b3d3229127b1ee9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f925d30573ec4495b3d3229127b1ee9b. Latest consensus state: current_term: 1 leader_uuid: "f925d30573ec4495b3d3229127b1ee9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f925d30573ec4495b3d3229127b1ee9b" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37821 } } }
I20260810 07:25:26.624193 22327 sys_catalog.cc:458] T 00000000000000000000000000000000 P f925d30573ec4495b3d3229127b1ee9b [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:26.624541 22326 sys_catalog.cc:455] T 00000000000000000000000000000000 P f925d30573ec4495b3d3229127b1ee9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f925d30573ec4495b3d3229127b1ee9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f925d30573ec4495b3d3229127b1ee9b" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37821 } } }
I20260810 07:25:26.624594 22339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:25:26.624680 22326 sys_catalog.cc:458] T 00000000000000000000000000000000 P f925d30573ec4495b3d3229127b1ee9b [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:26.628202 22339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:25:26.635118 22339 catalog_manager.cc:1347] Generated new cluster ID: 32cfccf3d1c843b09d568072a51ef586
I20260810 07:25:26.635242 22339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:25:26.661386 22339 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:25:26.662837 22339 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:25:26.680508 22339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f925d30573ec4495b3d3229127b1ee9b: Generated new TSK 0
I20260810 07:25:26.681433 22339 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
I20260810 07:25:26.726485 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:0
--local_ip_for_outbound_sockets=127.12.157.193
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37821
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:38007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:26.847314 22363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:26.847610 22363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:26.847659 22363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:26.852062 22363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:26.852144 22363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:26.852206 22363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:25:26.857751 22363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:38007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:26.858851 22363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:26.859875 22363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:26.869433 22382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:26.869467 22380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:26.869490 22379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:26.869669 22363 server_base.cc:1034] running on GCE node
I20260810 07:25:26.870224 22363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:26.870915 22363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:26.872126 22363 hybrid_clock.cc:648] HybridClock initialized: now 1786346726872111 us; error 32 us; skew 500 ppm
Aug 10 07:25:26 dist-test-slave-ffrd krb5kdc[22189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346726, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:26.875267 22363 init.cc:377] Logged in from keytab as kudu/127.12.157.193@KRBTEST.COM (short username kudu)
I20260810 07:25:26.876209 22363 webserver.cc:460] Webserver started at http://127.12.157.193:42461/ using document root <none> and password file <none>
I20260810 07:25:26.876796 22363 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:26.876895 22363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:26.877166 22363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:26.879245 22363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-0/data/instance:
uuid: "b4b7c3620de047a995861ec8c0fe23ab"
format_stamp: "Formatted at 2026-08-10 07:25:26 on dist-test-slave-ffrd"
server_key: "84f31b8587c51ed91e5d94ce9ab4eae2"
server_key_iv: "8c54c2a873d3e27151e37d01bb3ff544"
server_key_version: "encryptionkey@0"
I20260810 07:25:26.879842 22363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance:
uuid: "b4b7c3620de047a995861ec8c0fe23ab"
format_stamp: "Formatted at 2026-08-10 07:25:26 on dist-test-slave-ffrd"
server_key: "84f31b8587c51ed91e5d94ce9ab4eae2"
server_key_iv: "8c54c2a873d3e27151e37d01bb3ff544"
server_key_version: "encryptionkey@0"
I20260810 07:25:26.883961 22363 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260810 07:25:26.886536 22391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:26.887781 22363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 07:25:26.887931 22363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "b4b7c3620de047a995861ec8c0fe23ab"
format_stamp: "Formatted at 2026-08-10 07:25:26 on dist-test-slave-ffrd"
server_key: "84f31b8587c51ed91e5d94ce9ab4eae2"
server_key_iv: "8c54c2a873d3e27151e37d01bb3ff544"
server_key_version: "encryptionkey@0"
I20260810 07:25:26.888046 22363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:26.898393 22363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:26.901307 22363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:26.901515 22363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:26.902113 22363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:26.902956 22363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:26.903028 22363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:26.903100 22363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:26.903146 22363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:26.912328 22363 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:43827
I20260810 07:25:26.912359 22590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:43827 every 8 connection(s)
I20260810 07:25:26.913311 22363 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:25:26.915207 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 22363
I20260810 07:25:26.915426 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance
I20260810 07:25:26.915858 12919 external_mini_cluster.cc:1428] Setting key aed931afadef34f33477bee4b09ec0c8
Aug 10 07:25:26 dist-test-slave-ffrd krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346726, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.157.194@KRBTEST.COM; defaulting to no policy
I20260810 07:25:26.929466 22323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:26.915222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:59853 (local address 127.12.157.254:37821)
0810 07:25:26.915374 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:26.915379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:26.916503 (+ 1124us) server_negotiation.cc:407] Connection header received
0810 07:25:26.918257 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:26.918260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:26.918315 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:26.918399 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:26.919648 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:26.920196 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:26.921473 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:26.921706 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:26.921820 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:26.924721 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:26.924726 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:26.924730 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:26.924777 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:26.927051 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:26.927768 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:26.927773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:26.927774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:26.927871 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:26.928238 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:26.928241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:26.928243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:26.928451 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:26.928562 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:26.929144 (+ 582us) server_negotiation.cc:299] Negotiation successful
0810 07:25:26.929280 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 07:25:26.930358 22596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:26.915747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37821 (local address 127.12.157.193:59853)
0810 07:25:26.916316 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:26.916378 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:26.918042 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:26.918584 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:26.918592 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:26.918972 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:26.919496 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:26.919510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:26.920329 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:26.920332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:26.921320 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:26.921331 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:26.921866 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:26.921870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:26.922051 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:26.922701 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:26.922728 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:26.924422 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:26.927194 (+ 2772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:26.927201 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:26.927236 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:26.927632 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:26.927994 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:26.927996 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:26.927997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:26.928136 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:26.928562 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:26.928571 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:26.928844 (+ 273us) client_negotiation.cc:769] Sending connection context
0810 07:25:26.929101 (+ 257us) client_negotiation.cc:240] Negotiation successful
0810 07:25:26.929456 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":161,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.194@KRBTEST.COM" created.
I20260810 07:25:26.931770 22591 heartbeater.cc:347] Connected to a master server at 127.12.157.254:37821
I20260810 07:25:26.932029 22591 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:26.932523 22591 heartbeater.cc:511] Master 127.12.157.254:37821 requested a full tablet report, sending...
I20260810 07:25:26.934193 22263 ts_manager.cc:194] Registered new tserver with Master: b4b7c3620de047a995861ec8c0fe23ab (127.12.157.193:43827)
I20260810 07:25:26.935352 22263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.193@KRBTEST.COM'} at 127.12.157.193:59853
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
I20260810 07:25:26.951954 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:0
--local_ip_for_outbound_sockets=127.12.157.194
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37821
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:38007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:27.068771 22599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:27.069104 22599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:27.069156 22599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:27.073484 22599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:27.073570 22599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:27.073657 22599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:25:27.079483 22599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:38007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:27.080611 22599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:27.081748 22599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:27.092326 22615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:27.092379 22618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:27.092325 22613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:27.092803 22599 server_base.cc:1034] running on GCE node
I20260810 07:25:27.093283 22599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:27.094012 22599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:27.095228 22599 hybrid_clock.cc:648] HybridClock initialized: now 1786346727095210 us; error 30 us; skew 500 ppm
Aug 10 07:25:27 dist-test-slave-ffrd krb5kdc[22189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346727, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:27.098287 22599 init.cc:377] Logged in from keytab as kudu/127.12.157.194@KRBTEST.COM (short username kudu)
I20260810 07:25:27.099331 22599 webserver.cc:460] Webserver started at http://127.12.157.194:41083/ using document root <none> and password file <none>
I20260810 07:25:27.099910 22599 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:27.099998 22599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:27.100230 22599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:27.102386 22599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-1/data/instance:
uuid: "b33ce1c347b74653ab952daa06000904"
format_stamp: "Formatted at 2026-08-10 07:25:27 on dist-test-slave-ffrd"
server_key: "62babc63ec87295cb8b392bf12f725e6"
server_key_iv: "1877e7f648bf8601a38610620debe84a"
server_key_version: "encryptionkey@0"
I20260810 07:25:27.102952 22599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance:
uuid: "b33ce1c347b74653ab952daa06000904"
format_stamp: "Formatted at 2026-08-10 07:25:27 on dist-test-slave-ffrd"
server_key: "62babc63ec87295cb8b392bf12f725e6"
server_key_iv: "1877e7f648bf8601a38610620debe84a"
server_key_version: "encryptionkey@0"
I20260810 07:25:27.106808 22599 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260810 07:25:27.109567 22627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:27.110872 22599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:25:27.111022 22599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "b33ce1c347b74653ab952daa06000904"
format_stamp: "Formatted at 2026-08-10 07:25:27 on dist-test-slave-ffrd"
server_key: "62babc63ec87295cb8b392bf12f725e6"
server_key_iv: "1877e7f648bf8601a38610620debe84a"
server_key_version: "encryptionkey@0"
I20260810 07:25:27.111140 22599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:27.137506 22599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:27.140754 22599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:27.141021 22599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:27.141676 22599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:27.142607 22599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:27.142670 22599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:27.142740 22599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:27.142786 22599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:27.152109 22599 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:45779
I20260810 07:25:27.152137 22837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:45779 every 8 connection(s)
I20260810 07:25:27.153111 22599 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:25:27.159909 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 22599
I20260810 07:25:27.160006 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance
I20260810 07:25:27.160275 12919 external_mini_cluster.cc:1428] Setting key 48909649c6ad03769299b89538dd0fcc
Aug 10 07:25:27 dist-test-slave-ffrd krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346727, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:27.167979 22323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:27.154930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:40755 (local address 127.12.157.254:37821)
0810 07:25:27.155081 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:27.155085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:27.155877 (+ 792us) server_negotiation.cc:407] Connection header received
0810 07:25:27.157437 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:27.157441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:27.157503 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:27.157606 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:27.158814 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:27.159356 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:27.160666 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:27.160969 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:27.161078 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:27.163637 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:27.163641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:27.163644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:27.163675 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:27.165527 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:27.166177 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:27.166180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:27.166181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:27.166266 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:27.166690 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:27.166693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:27.166695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:27.166913 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:27.167003 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:27.167734 (+ 731us) server_negotiation.cc:299] Negotiation successful
0810 07:25:27.167851 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 07:25:27.169293 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:27.155242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37821 (local address 127.12.157.194:40755)
0810 07:25:27.155718 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:27.155762 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:27.157196 (+ 1434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:27.157777 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:27.157784 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:27.158162 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:27.158660 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:27.158674 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:27.159491 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:27.159494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:27.160544 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:27.160554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:27.161130 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:27.161134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:27.161329 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:27.162022 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:27.162039 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:27.163480 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:27.165651 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:27.165660 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:27.165680 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:27.166062 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:27.166365 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:27.166370 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:27.166375 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:27.166571 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:27.167028 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:27.167037 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:27.167418 (+ 381us) client_negotiation.cc:769] Sending connection context
0810 07:25:27.167655 (+ 237us) client_negotiation.cc:240] Negotiation successful
0810 07:25:27.168068 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":113,"threads_started":1}
I20260810 07:25:27.170661 22838 heartbeater.cc:347] Connected to a master server at 127.12.157.254:37821
I20260810 07:25:27.170948 22838 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:27.171568 22838 heartbeater.cc:511] Master 127.12.157.254:37821 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.157.195@KRBTEST.COM; defaulting to no policy
I20260810 07:25:27.172920 22263 ts_manager.cc:194] Registered new tserver with Master: b33ce1c347b74653ab952daa06000904 (127.12.157.194:45779)
I20260810 07:25:27.173696 22263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.194@KRBTEST.COM'} at 127.12.157.194:40755
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
I20260810 07:25:27.192476 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:0
--local_ip_for_outbound_sockets=127.12.157.195
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37821
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:38007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:27.310024 22849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:27.310336 22849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:27.310410 22849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:27.314519 22849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:27.314600 22849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:27.314687 22849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:25:27.320225 22849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:38007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:27.321409 22849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:27.322427 22849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:27.333359 22860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:27.333372 22865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:27.333467 22863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:27.333804 22849 server_base.cc:1034] running on GCE node
I20260810 07:25:27.334177 22849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:27.334815 22849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:27.336040 22849 hybrid_clock.cc:648] HybridClock initialized: now 1786346727336025 us; error 36 us; skew 500 ppm
Aug 10 07:25:27 dist-test-slave-ffrd krb5kdc[22189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346727, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:27.338922 22849 init.cc:377] Logged in from keytab as kudu/127.12.157.195@KRBTEST.COM (short username kudu)
I20260810 07:25:27.339788 22849 webserver.cc:460] Webserver started at http://127.12.157.195:37945/ using document root <none> and password file <none>
I20260810 07:25:27.340305 22849 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:27.340363 22849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:27.340533 22849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:27.342628 22849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-2/data/instance:
uuid: "02666bc271db47289caccbddcd0e1c07"
format_stamp: "Formatted at 2026-08-10 07:25:27 on dist-test-slave-ffrd"
server_key: "921d9ecc3d9d133de5a8cb32a5a68c2e"
server_key_iv: "e0938e4b26b6b0e9897ed771b8cfb0a5"
server_key_version: "encryptionkey@0"
I20260810 07:25:27.343186 22849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance:
uuid: "02666bc271db47289caccbddcd0e1c07"
format_stamp: "Formatted at 2026-08-10 07:25:27 on dist-test-slave-ffrd"
server_key: "921d9ecc3d9d133de5a8cb32a5a68c2e"
server_key_iv: "e0938e4b26b6b0e9897ed771b8cfb0a5"
server_key_version: "encryptionkey@0"
I20260810 07:25:27.347054 22849 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260810 07:25:27.349576 22878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:27.350811 22849 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:25:27.350922 22849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "02666bc271db47289caccbddcd0e1c07"
format_stamp: "Formatted at 2026-08-10 07:25:27 on dist-test-slave-ffrd"
server_key: "921d9ecc3d9d133de5a8cb32a5a68c2e"
server_key_iv: "e0938e4b26b6b0e9897ed771b8cfb0a5"
server_key_version: "encryptionkey@0"
I20260810 07:25:27.351011 22849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:27.362566 22849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:27.365592 22849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:27.365744 22849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:27.366283 22849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:27.367063 22849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:27.367112 22849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:27.367152 22849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:27.367177 22849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:27.376518 22849 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:40207
I20260810 07:25:27.376546 23071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:40207 every 8 connection(s)
I20260810 07:25:27.377452 22849 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:25:27.380000 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 22849
I20260810 07:25:27.380117 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance
I20260810 07:25:27.380419 12919 external_mini_cluster.cc:1428] Setting key b837b4e617b73917cf82e1188f8ca604
Aug 10 07:25:27 dist-test-slave-ffrd krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346727, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:27.390861 22323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:27.379115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:43543 (local address 127.12.157.254:37821)
0810 07:25:27.379243 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:27.379247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:27.380097 (+ 850us) server_negotiation.cc:407] Connection header received
0810 07:25:27.381706 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:27.381709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:27.381773 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:27.381862 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:27.383003 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:27.383534 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:27.384614 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:27.384821 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:27.384896 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:27.387330 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:27.387334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:27.387336 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:27.387368 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:27.388703 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:27.389263 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:27.389266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:27.389267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:27.389347 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:27.389740 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:27.389743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:27.389744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:27.389919 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:27.389999 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:27.390606 (+ 607us) server_negotiation.cc:299] Negotiation successful
0810 07:25:27.390735 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 07:25:27.391687 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:27.379409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37821 (local address 127.12.157.195:43543)
0810 07:25:27.379904 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:27.379964 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:27.381481 (+ 1517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:27.382009 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:27.382015 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:27.382357 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:27.382871 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:27.382886 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:27.383639 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:27.383641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:27.384508 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:27.384517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:27.384982 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:27.384991 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:27.385135 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:27.385719 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:27.385736 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:27.387176 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:27.388816 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:27.388823 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:27.388843 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:27.389178 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:27.389466 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:27.389469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:27.389470 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:27.389633 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:27.390013 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:27.390019 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:27.390257 (+ 238us) client_negotiation.cc:769] Sending connection context
0810 07:25:27.390438 (+ 181us) client_negotiation.cc:240] Negotiation successful
0810 07:25:27.390754 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":118,"threads_started":1}
I20260810 07:25:27.392997 23072 heartbeater.cc:347] Connected to a master server at 127.12.157.254:37821
I20260810 07:25:27.393263 23072 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:27.393939 23072 heartbeater.cc:511] Master 127.12.157.254:37821 requested a full tablet report, sending...
I20260810 07:25:27.395103 22263 ts_manager.cc:194] Registered new tserver with Master: 02666bc271db47289caccbddcd0e1c07 (127.12.157.195:40207)
I20260810 07:25:27.395758 22263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.195@KRBTEST.COM'} at 127.12.157.195:43543
I20260810 07:25:27.406177 12919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/test-admin.keytab.
Aug 10 07:25:27 dist-test-slave-ffrd krb5kdc[22189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:27.423521 12919 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 07:25:27.426550 23083 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:27.425114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45960 (local address 127.12.157.193:43827)
0810 07:25:27.425426 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:27.425442 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:27.425466 (+ 24us) server_negotiation.cc:407] Connection header received
0810 07:25:27.425534 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:27.425539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:27.425724 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:27.425841 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:27.426335 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45960: BlockingRecv error: recv got EOF from 127.0.0.1:45960 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":104,"threads_started":1}
W20260810 07:25:27.426600 12919 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/krb5cc' of type FILE
Aug 10 07:25:27 dist-test-slave-ffrd krb5kdc[22189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:27.427912 12919 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 07:25:27 dist-test-slave-ffrd krb5kdc[22189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:27.428865 12919 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 07:25:27 dist-test-slave-ffrd krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.193@KRBTEST.COM
I20260810 07:25:27.438836 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:27.429765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45964 (local address 127.12.157.193:43827)
0810 07:25:27.429927 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:27.429931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:27.429991 (+ 60us) server_negotiation.cc:407] Connection header received
0810 07:25:27.430107 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:27.430109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:27.430147 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:27.430216 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:27.430759 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:27.431531 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:27.432381 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:27.432842 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:27.432994 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:27.435536 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:27.435547 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:27.435560 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:27.435585 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:27.437191 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:27.437606 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:27.437612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:27.437613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:27.437696 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:27.437931 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:27.437933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:27.437934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:27.438175 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:27.438340 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:27.438497 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 07:25:27.438671 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260810 07:25:27.442852 23083 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:27.441860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45976 (local address 127.12.157.193:43827)
0810 07:25:27.441981 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:27.441985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:27.442097 (+ 112us) server_negotiation.cc:407] Connection header received
0810 07:25:27.442194 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:27.442200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:27.442246 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:27.442324 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:27.442737 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45976: BlockingRecv error: recv got EOF from 127.0.0.1:45976 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 07:25:27.442909 12919 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/krb5cc' of type FILE
Aug 10 07:25:27 dist-test-slave-ffrd krb5kdc[22189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:27.443948 12919 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 07:25:27 dist-test-slave-ffrd krb5kdc[22189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:27.444897 12919 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 07:25:27 dist-test-slave-ffrd krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.193@KRBTEST.COM
I20260810 07:25:27.453147 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:27.445759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45980 (local address 127.12.157.193:43827)
0810 07:25:27.446002 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:27.446005 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:27.446017 (+ 12us) server_negotiation.cc:407] Connection header received
0810 07:25:27.446046 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:27.446048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:27.446078 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:27.446142 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:27.446606 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:27.447115 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:27.447906 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:27.448154 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:27.448238 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:27.450335 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:27.450338 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:27.450340 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:27.450372 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:27.451570 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:27.451948 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:27.451951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:27.451952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:27.452035 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:27.452295 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:27.452298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:27.452299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:27.452487 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:27.452609 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:27.452811 (+ 202us) server_negotiation.cc:299] Negotiation successful
0810 07:25:27.452943 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
W20260810 07:25:27.455839 23083 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:27.454884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45984 (local address 127.12.157.193:43827)
0810 07:25:27.455005 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:27.455009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:27.455104 (+ 95us) server_negotiation.cc:407] Connection header received
0810 07:25:27.455234 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:27.455237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:27.455277 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:27.455362 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:27.455739 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45984: BlockingRecv error: recv got EOF from 127.0.0.1:45984 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260810 07:25:27.455960 12919 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786346694975122-12919-0/krb5kdc/krb5cc' of type FILE
Aug 10 07:25:27 dist-test-slave-ffrd krb5kdc[22189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:27.457109 12919 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 07:25:27 dist-test-slave-ffrd krb5kdc[22189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:27.457952 12919 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 07:25:27 dist-test-slave-ffrd krb5kdc[22189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.193@KRBTEST.COM
I20260810 07:25:27.466480 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:27.458710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45992 (local address 127.12.157.193:43827)
0810 07:25:27.458859 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:27.458869 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:27.458884 (+ 15us) server_negotiation.cc:407] Connection header received
0810 07:25:27.458977 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:27.458980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:27.459016 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:27.459089 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:27.459524 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:27.460050 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:27.461074 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:27.461545 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:27.461659 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:27.463845 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:27.463849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:27.463851 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:27.463873 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:27.465097 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:27.465500 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:27.465503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:27.465504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:27.465573 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:27.465800 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:27.465802 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:27.465803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:27.465950 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:27.466063 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:27.466228 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 07:25:27.466308 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 07:25:27.467391 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 22363
I20260810 07:25:27.490114 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 22599
I20260810 07:25:27.502172 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 22849
I20260810 07:25:27.515389 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 22208
2026-08-10T07:25:27Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2548 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:25:27 dist-test-slave-ffrd krb5kdc[23110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:25:27 dist-test-slave-ffrd krb5kdc[23110](info): set up 2 sockets
Aug 10 07:25:27 dist-test-slave-ffrd krb5kdc[23110](info): commencing operation
krb5kdc: starting...
W20260810 07:25:28.809221 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.269s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:25:28 dist-test-slave-ffrd krb5kdc[23110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:25:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:25:28Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.254.keytab.
Entry for principal oryx/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.254.keytab.
I20260810 07:25:28.948079 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:46027
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.254.keytab
--principal=oryx/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:43127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:46027
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:29.067013 23137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:29.067286 23137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:29.067327 23137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:29.072451 23137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:25:29.072522 23137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:29.072542 23137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:29.072559 23137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:25:29.072574 23137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:25:29.078011 23137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:43127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:46027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.254.keytab
--principal=oryx/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:46027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:29.079099 23137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:29.080091 23137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:29.088840 23149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:29.088887 23146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:29.088879 23144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:29.089169 23137 server_base.cc:1034] running on GCE node
I20260810 07:25:29.089864 23137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:29.090934 23137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:29.092105 23137 hybrid_clock.cc:648] HybridClock initialized: now 1786346729092076 us; error 40 us; skew 500 ppm
Aug 10 07:25:29 dist-test-slave-ffrd krb5kdc[23110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346729, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:29.095103 23137 init.cc:377] Logged in from keytab as oryx/127.12.157.254@KRBTEST.COM (short username oryx)
I20260810 07:25:29.096030 23137 webserver.cc:460] Webserver started at http://127.12.157.254:36123/ using document root <none> and password file <none>
I20260810 07:25:29.096611 23137 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:29.096704 23137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:29.096982 23137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:29.099078 23137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "eebcdb40543d45c5aef6817f84cef282"
format_stamp: "Formatted at 2026-08-10 07:25:29 on dist-test-slave-ffrd"
server_key: "7ab430acef7a7ef08009061cc127daf2"
server_key_iv: "bbf980d7bd901b33210f8247b6a470b9"
server_key_version: "encryptionkey@0"
I20260810 07:25:29.099675 23137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "eebcdb40543d45c5aef6817f84cef282"
format_stamp: "Formatted at 2026-08-10 07:25:29 on dist-test-slave-ffrd"
server_key: "7ab430acef7a7ef08009061cc127daf2"
server_key_iv: "bbf980d7bd901b33210f8247b6a470b9"
server_key_version: "encryptionkey@0"
I20260810 07:25:29.103520 23137 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 07:25:29.106127 23163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:29.107396 23137 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:25:29.107547 23137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "eebcdb40543d45c5aef6817f84cef282"
format_stamp: "Formatted at 2026-08-10 07:25:29 on dist-test-slave-ffrd"
server_key: "7ab430acef7a7ef08009061cc127daf2"
server_key_iv: "bbf980d7bd901b33210f8247b6a470b9"
server_key_version: "encryptionkey@0"
I20260810 07:25:29.107664 23137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:29.135591 23137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:29.139154 23137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:29.139402 23137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:29.146813 23137 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:46027
I20260810 07:25:29.147132 23246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:46027 every 8 connection(s)
I20260810 07:25:29.147858 23137 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:25:29.150842 23249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:29.155866 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 23137
I20260810 07:25:29.156024 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:25:29.156324 12919 external_mini_cluster.cc:1428] Setting key 509e1a86c55054daaa232c36eb0df0d8
I20260810 07:25:29.156980 23249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282: Bootstrap starting.
I20260810 07:25:29.159647 23249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:29.160430 23249 log.cc:824] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:29.162413 23249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282: No bootstrap required, opened a new log
Aug 10 07:25:29 dist-test-slave-ffrd krb5kdc[23110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM
I20260810 07:25:29.166738 23249 raft_consensus.cc:348] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } }
I20260810 07:25:29.167002 23249 raft_consensus.cc:374] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:29.167088 23249 raft_consensus.cc:729] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eebcdb40543d45c5aef6817f84cef282, State: Initialized, Role: FOLLOWER
I20260810 07:25:29.167591 23249 consensus_queue.cc:260] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } }
I20260810 07:25:29.167724 23249 raft_consensus.cc:388] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:25:29.167819 23249 raft_consensus.cc:482] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:25:29.167948 23249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:29.169135 23249 raft_consensus.cc:504] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } }
I20260810 07:25:29.169531 23249 leader_election.cc:302] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eebcdb40543d45c5aef6817f84cef282; no voters:
I20260810 07:25:29.169884 23249 leader_election.cc:288] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:25:29.169958 23254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:29.170252 23254 raft_consensus.cc:686] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [term 1 LEADER]: Becoming Leader. State: Replica: eebcdb40543d45c5aef6817f84cef282, State: Running, Role: LEADER
I20260810 07:25:29.170614 23254 consensus_queue.cc:237] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } }
I20260810 07:25:29.171315 23249 sys_catalog.cc:564] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:25:29.171607 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:29.157899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52568 (local address 127.12.157.254:46027)
0810 07:25:29.158412 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:29.158431 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:29.158466 (+ 35us) server_negotiation.cc:407] Connection header received
0810 07:25:29.159110 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:29.159127 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:29.159460 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:29.159797 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:29.160631 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:29.161865 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:29.162764 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:29.163226 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:29.163427 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:29.166272 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:29.166277 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:29.166290 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:29.166325 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:29.168291 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:29.168761 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:29.168766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:29.168770 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:29.168875 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:29.169222 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:29.169225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:29.169226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:29.169834 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:29.170040 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:29.170375 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 07:25:29.170826 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":174,"threads_started":1}
I20260810 07:25:29.171826 23254 sys_catalog.cc:455] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eebcdb40543d45c5aef6817f84cef282. Latest consensus state: current_term: 1 leader_uuid: "eebcdb40543d45c5aef6817f84cef282" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } }
I20260810 07:25:29.171942 23254 sys_catalog.cc:458] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:29.172151 23254 sys_catalog.cc:455] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eebcdb40543d45c5aef6817f84cef282" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } }
I20260810 07:25:29.172247 23254 sys_catalog.cc:458] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:29.174077 23263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:25:29.176900 23263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:25:29.182449 23263 catalog_manager.cc:1347] Generated new cluster ID: 45c8a83eed7741dfa445f32c118347c1
I20260810 07:25:29.182521 23263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:25:29.206872 23263 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:25:29.207980 23263 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:25:29.212034 23263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282: Generated new TSK 0
I20260810 07:25:29.213059 23263 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.193.keytab.
Entry for principal oryx/127.12.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.193.keytab.
I20260810 07:25:29.255654 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:0
--local_ip_for_outbound_sockets=127.12.157.193
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:46027
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.193.keytab
--principal=oryx/127.12.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:43127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:29.375705 23287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:29.376060 23287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:29.376142 23287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:29.380430 23287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:29.380530 23287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:29.380628 23287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:25:29.386252 23287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:43127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.193.keytab
--principal=oryx/127.12.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:46027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:29.387419 23287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:29.388509 23287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:29.398630 23294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:29.398638 23295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:29.398630 23299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:29.398955 23287 server_base.cc:1034] running on GCE node
I20260810 07:25:29.399355 23287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:29.399968 23287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:29.401118 23287 hybrid_clock.cc:648] HybridClock initialized: now 1786346729401101 us; error 30 us; skew 500 ppm
Aug 10 07:25:29 dist-test-slave-ffrd krb5kdc[23110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346729, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:29.404026 23287 init.cc:377] Logged in from keytab as oryx/127.12.157.193@KRBTEST.COM (short username oryx)
I20260810 07:25:29.405020 23287 webserver.cc:460] Webserver started at http://127.12.157.193:46827/ using document root <none> and password file <none>
I20260810 07:25:29.405594 23287 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:29.405687 23287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:29.405952 23287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:29.407997 23287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/data/instance:
uuid: "373169c708e84b2c9a7e52676b00de91"
format_stamp: "Formatted at 2026-08-10 07:25:29 on dist-test-slave-ffrd"
server_key: "947b8081c1d4dd4837a903d85eccf68e"
server_key_iv: "e0ecf3ec8af1199b39a45a18d6da456a"
server_key_version: "encryptionkey@0"
I20260810 07:25:29.408571 23287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance:
uuid: "373169c708e84b2c9a7e52676b00de91"
format_stamp: "Formatted at 2026-08-10 07:25:29 on dist-test-slave-ffrd"
server_key: "947b8081c1d4dd4837a903d85eccf68e"
server_key_iv: "e0ecf3ec8af1199b39a45a18d6da456a"
server_key_version: "encryptionkey@0"
I20260810 07:25:29.412528 23287 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 07:25:29.415187 23308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:29.416414 23287 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 07:25:29.416553 23287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "373169c708e84b2c9a7e52676b00de91"
format_stamp: "Formatted at 2026-08-10 07:25:29 on dist-test-slave-ffrd"
server_key: "947b8081c1d4dd4837a903d85eccf68e"
server_key_iv: "e0ecf3ec8af1199b39a45a18d6da456a"
server_key_version: "encryptionkey@0"
I20260810 07:25:29.416672 23287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:29.440748 23287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:29.444062 23287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:29.444276 23287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:29.444929 23287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:29.445848 23287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:29.445921 23287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:29.445999 23287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:29.446043 23287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:29.455497 23287 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:36441
I20260810 07:25:29.455530 23485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:36441 every 8 connection(s)
I20260810 07:25:29.456386 23287 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:25:29.464308 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 23287
I20260810 07:25:29.464445 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance
I20260810 07:25:29.464732 12919 external_mini_cluster.cc:1428] Setting key be51aaabebfef7621d8329f274e6dca4
Aug 10 07:25:29 dist-test-slave-ffrd krb5kdc[23110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346729, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM
I20260810 07:25:29.469997 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:29.458180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:35741 (local address 127.12.157.254:46027)
0810 07:25:29.458352 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:29.458357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:29.459165 (+ 808us) server_negotiation.cc:407] Connection header received
0810 07:25:29.460516 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:29.460519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:29.460565 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:29.460641 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:29.461844 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:29.462396 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:29.463524 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:29.463747 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:29.463848 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:29.466493 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:29.466510 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:29.466512 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:29.466535 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:29.467901 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:29.468423 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:29.468426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:29.468430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:29.468511 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:29.468860 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:29.468865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:29.468866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:29.469104 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:29.469214 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:29.469765 (+ 551us) server_negotiation.cc:299] Negotiation successful
0810 07:25:29.469877 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 07:25:29.471118 23491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:29.458551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.12.157.193:35741)
0810 07:25:29.459014 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:29.459056 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:29.460317 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:29.460802 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:29.460819 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:29.461244 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:29.461703 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:29.461716 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:29.462517 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:29.462520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:29.463413 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:29.463423 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:29.463890 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:29.463893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:29.464028 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:29.464669 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:29.464686 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:29.466323 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:29.468029 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:29.468035 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:29.468048 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:29.468312 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:29.468610 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:29.468614 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:29.468616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:29.468761 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:29.469225 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:29.469233 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:29.469478 (+ 245us) client_negotiation.cc:769] Sending connection context
0810 07:25:29.469707 (+ 229us) client_negotiation.cc:240] Negotiation successful
0810 07:25:29.470059 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":103,"threads_started":1}
I20260810 07:25:29.472543 23488 heartbeater.cc:347] Connected to a master server at 127.12.157.254:46027
I20260810 07:25:29.472826 23488 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:29.473490 23488 heartbeater.cc:511] Master 127.12.157.254:46027 requested a full tablet report, sending...
I20260810 07:25:29.475133 23186 ts_manager.cc:194] Registered new tserver with Master: 373169c708e84b2c9a7e52676b00de91 (127.12.157.193:36441)
WARNING: no policy specified for oryx/127.12.157.194@KRBTEST.COM; defaulting to no policy
I20260810 07:25:29.476373 23186 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.157.193@KRBTEST.COM'} at 127.12.157.193:35741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.194.keytab.
Entry for principal oryx/127.12.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.194.keytab.
I20260810 07:25:29.497705 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:0
--local_ip_for_outbound_sockets=127.12.157.194
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:46027
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.194.keytab
--principal=oryx/127.12.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:43127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:29.612275 23494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:29.612558 23494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:29.612629 23494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:29.616612 23494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:29.616711 23494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:29.616811 23494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:25:29.622352 23494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:43127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.194.keytab
--principal=oryx/127.12.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:46027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:29.623471 23494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:29.624523 23494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:29.634924 23501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:29.634989 23500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:29.635193 23494 server_base.cc:1034] running on GCE node
W20260810 07:25:29.634946 23503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:29.635999 23494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:29.636691 23494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:29.637876 23494 hybrid_clock.cc:648] HybridClock initialized: now 1786346729637866 us; error 63 us; skew 500 ppm
Aug 10 07:25:29 dist-test-slave-ffrd krb5kdc[23110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346729, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:29.640883 23494 init.cc:377] Logged in from keytab as oryx/127.12.157.194@KRBTEST.COM (short username oryx)
I20260810 07:25:29.641867 23494 webserver.cc:460] Webserver started at http://127.12.157.194:35303/ using document root <none> and password file <none>
I20260810 07:25:29.642428 23494 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:29.642488 23494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:29.642660 23494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:29.645056 23494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/data/instance:
uuid: "313322cdeb7048778cb6dedb8a4aa76e"
format_stamp: "Formatted at 2026-08-10 07:25:29 on dist-test-slave-ffrd"
server_key: "e81d3b6b388e1c68bda448c271b84edf"
server_key_iv: "95b2430bfffee35e45f7888e74708f66"
server_key_version: "encryptionkey@0"
I20260810 07:25:29.645640 23494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance:
uuid: "313322cdeb7048778cb6dedb8a4aa76e"
format_stamp: "Formatted at 2026-08-10 07:25:29 on dist-test-slave-ffrd"
server_key: "e81d3b6b388e1c68bda448c271b84edf"
server_key_iv: "95b2430bfffee35e45f7888e74708f66"
server_key_version: "encryptionkey@0"
I20260810 07:25:29.649636 23494 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 07:25:29.652287 23516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:29.653755 23494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 07:25:29.653903 23494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "313322cdeb7048778cb6dedb8a4aa76e"
format_stamp: "Formatted at 2026-08-10 07:25:29 on dist-test-slave-ffrd"
server_key: "e81d3b6b388e1c68bda448c271b84edf"
server_key_iv: "95b2430bfffee35e45f7888e74708f66"
server_key_version: "encryptionkey@0"
I20260810 07:25:29.654021 23494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:29.675535 23494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:29.679069 23494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:29.679287 23494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:29.679942 23494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:29.680831 23494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:29.680881 23494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:29.680953 23494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:29.681016 23494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:29.690575 23494 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:37871
I20260810 07:25:29.690598 23699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:37871 every 8 connection(s)
I20260810 07:25:29.691514 23494 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:25:29.695289 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 23494
I20260810 07:25:29.695586 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance
I20260810 07:25:29.695914 12919 external_mini_cluster.cc:1428] Setting key c237114112a43642978e62e85b9264f5
Aug 10 07:25:29 dist-test-slave-ffrd krb5kdc[23110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346729, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.12.157.195@KRBTEST.COM; defaulting to no policy
I20260810 07:25:29.707582 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:29.693540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:53853 (local address 127.12.157.254:46027)
0810 07:25:29.693697 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:29.693700 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:29.694647 (+ 947us) server_negotiation.cc:407] Connection header received
0810 07:25:29.696153 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:29.696157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:29.696216 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:29.696332 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:29.697893 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:29.698440 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:29.700149 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:29.700556 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:29.700738 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:29.703448 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:29.703455 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:29.703460 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:29.703499 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:29.705233 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:29.705813 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:29.705819 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:29.705822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:29.705932 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:29.706284 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:29.706290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:29.706294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:29.706560 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:29.706677 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:29.707216 (+ 539us) server_negotiation.cc:299] Negotiation successful
0810 07:25:29.707370 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 07:25:29.708595 23707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:29.693922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.12.157.194:53853)
0810 07:25:29.694494 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:29.694540 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:29.695915 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:29.696555 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:29.696564 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:29.697031 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:29.697701 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:29.697721 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:29.698581 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:29.698588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:29.700011 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:29.700022 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:29.700716 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:29.700719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:29.700866 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:29.701697 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:29.701714 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:29.703251 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:29.705394 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:29.705400 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:29.705412 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:29.705697 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:29.706035 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:29.706037 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:29.706038 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:29.706179 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:29.706696 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:29.706701 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:29.706948 (+ 247us) client_negotiation.cc:769] Sending connection context
0810 07:25:29.707126 (+ 178us) client_negotiation.cc:240] Negotiation successful
0810 07:25:29.707540 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":170,"threads_started":1}
I20260810 07:25:29.710023 23700 heartbeater.cc:347] Connected to a master server at 127.12.157.254:46027
I20260810 07:25:29.710297 23700 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.157.195@KRBTEST.COM" created.
I20260810 07:25:29.710877 23700 heartbeater.cc:511] Master 127.12.157.254:46027 requested a full tablet report, sending...
I20260810 07:25:29.712145 23186 ts_manager.cc:194] Registered new tserver with Master: 313322cdeb7048778cb6dedb8a4aa76e (127.12.157.194:37871)
I20260810 07:25:29.712895 23186 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.157.194@KRBTEST.COM'} at 127.12.157.194:53853
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.195.keytab.
Entry for principal oryx/127.12.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.195.keytab.
I20260810 07:25:29.728552 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:0
--local_ip_for_outbound_sockets=127.12.157.195
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:46027
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.195.keytab
--principal=oryx/127.12.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:43127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:29.843910 23712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:29.844179 23712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:29.844256 23712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:29.848244 23712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:29.848341 23712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:29.848438 23712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:25:29.853722 23712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:43127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.195.keytab
--principal=oryx/127.12.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:46027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:29.854784 23712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:29.855803 23712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 07:25:29.865550 23712 server_base.cc:1034] running on GCE node
W20260810 07:25:29.865483 23720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:29.865478 23725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:29.865478 23722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:29.866179 23712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:29.866780 23712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:29.867995 23712 hybrid_clock.cc:648] HybridClock initialized: now 1786346729867979 us; error 26 us; skew 500 ppm
Aug 10 07:25:29 dist-test-slave-ffrd krb5kdc[23110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346729, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:29.870813 23712 init.cc:377] Logged in from keytab as oryx/127.12.157.195@KRBTEST.COM (short username oryx)
I20260810 07:25:29.871781 23712 webserver.cc:460] Webserver started at http://127.12.157.195:33401/ using document root <none> and password file <none>
I20260810 07:25:29.872450 23712 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:29.872546 23712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:29.872800 23712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:29.874909 23712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/data/instance:
uuid: "862aee2e55284192955041c46f55f8ea"
format_stamp: "Formatted at 2026-08-10 07:25:29 on dist-test-slave-ffrd"
server_key: "4415c2117a813753cd6c358458375783"
server_key_iv: "6c947acc330ddea6db4e2ecce54a1eec"
server_key_version: "encryptionkey@0"
I20260810 07:25:29.875481 23712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance:
uuid: "862aee2e55284192955041c46f55f8ea"
format_stamp: "Formatted at 2026-08-10 07:25:29 on dist-test-slave-ffrd"
server_key: "4415c2117a813753cd6c358458375783"
server_key_iv: "6c947acc330ddea6db4e2ecce54a1eec"
server_key_version: "encryptionkey@0"
I20260810 07:25:29.879220 23712 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 07:25:29.881683 23735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:29.882876 23712 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 07:25:29.883018 23712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "862aee2e55284192955041c46f55f8ea"
format_stamp: "Formatted at 2026-08-10 07:25:29 on dist-test-slave-ffrd"
server_key: "4415c2117a813753cd6c358458375783"
server_key_iv: "6c947acc330ddea6db4e2ecce54a1eec"
server_key_version: "encryptionkey@0"
I20260810 07:25:29.883131 23712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:29.908409 23712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:29.912132 23712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:29.912350 23712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:29.913048 23712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:29.913947 23712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:29.914021 23712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:29.914096 23712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:29.914149 23712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:29.923840 23712 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:32917
I20260810 07:25:29.923859 23921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:32917 every 8 connection(s)
I20260810 07:25:29.924784 23712 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:25:29.926470 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 23712
I20260810 07:25:29.926616 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance
I20260810 07:25:29.926952 12919 external_mini_cluster.cc:1428] Setting key 6e3fe83b50ab1d79e7461fae721d7da9
Aug 10 07:25:29 dist-test-slave-ffrd krb5kdc[23110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346729, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM
I20260810 07:25:29.940346 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:29.926751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:33057 (local address 127.12.157.254:46027)
0810 07:25:29.926929 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:29.926933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:29.927849 (+ 916us) server_negotiation.cc:407] Connection header received
0810 07:25:29.929346 (+ 1497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:29.929349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:29.929407 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:29.929485 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:29.930666 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:29.931212 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:29.932396 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:29.932637 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:29.932763 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:29.935536 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:29.935539 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:29.935541 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:29.935574 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:29.937489 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:29.938218 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:29.938223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:29.938226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:29.938336 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:29.938774 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:29.938778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:29.938780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:29.939028 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:29.939139 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:29.940017 (+ 878us) server_negotiation.cc:299] Negotiation successful
0810 07:25:29.940170 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 07:25:29.941510 23926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:29.927202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.12.157.195:33057)
0810 07:25:29.927694 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:29.927746 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:29.929130 (+ 1384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:29.929649 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:29.929656 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:29.930045 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:29.930522 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:29.930535 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:29.931363 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:29.931366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:29.932264 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:29.932274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:29.932815 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:29.932820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:29.933058 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:29.933792 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:29.933810 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:29.935339 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:29.937642 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:29.937648 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:29.937665 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:29.938082 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:29.938432 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:29.938436 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:29.938438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:29.938662 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:29.939184 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:29.939191 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:29.939692 (+ 501us) client_negotiation.cc:769] Sending connection context
0810 07:25:29.939958 (+ 266us) client_negotiation.cc:240] Negotiation successful
0810 07:25:29.940355 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":145,"threads_started":1}
I20260810 07:25:29.942541 23924 heartbeater.cc:347] Connected to a master server at 127.12.157.254:46027
I20260810 07:25:29.942819 23924 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:29.943336 23924 heartbeater.cc:511] Master 127.12.157.254:46027 requested a full tablet report, sending...
I20260810 07:25:29.944339 23186 ts_manager.cc:194] Registered new tserver with Master: 862aee2e55284192955041c46f55f8ea (127.12.157.195:32917)
I20260810 07:25:29.944926 23186 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.157.195@KRBTEST.COM'} at 127.12.157.195:33057
I20260810 07:25:29.954044 12919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.157.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.157.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.157.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.253.keytab.
Entry for principal oryx/127.12.157.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.253.keytab.
I20260810 07:25:29.984221 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.253:33175
--webserver_interface=127.12.157.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.253.keytab
--principal=oryx/127.12.157.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:43127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:46027,127.12.157.253:33175
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:30.110877 23930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:30.111210 23930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:30.111323 23930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:30.115897 23930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:25:30.115991 23930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:30.116032 23930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:30.116070 23930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:25:30.116102 23930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:25:30.121639 23930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:43127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:46027,127.12.157.253:33175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.253.keytab
--principal=oryx/127.12.157.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.253:33175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.157.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
W20260810 07:25:30.121902 23930 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.12.157.254:46027,127.12.157.253:33175'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 07:25:30.122826 23930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:30.123880 23930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:30.132419 23942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:30.132395 23940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:30.132540 23930 server_base.cc:1034] running on GCE node
W20260810 07:25:30.132372 23939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:30.133360 23930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:30.134470 23930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:30.135639 23930 hybrid_clock.cc:648] HybridClock initialized: now 1786346730135626 us; error 26 us; skew 500 ppm
Aug 10 07:25:30 dist-test-slave-ffrd krb5kdc[23110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346730, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:30.138803 23930 init.cc:377] Logged in from keytab as oryx/127.12.157.253@KRBTEST.COM (short username oryx)
I20260810 07:25:30.139724 23930 webserver.cc:460] Webserver started at http://127.12.157.253:43853/ using document root <none> and password file <none>
I20260810 07:25:30.140317 23930 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:30.140384 23930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:30.140612 23930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:30.142911 23930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-1/data/instance:
uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf"
format_stamp: "Formatted at 2026-08-10 07:25:30 on dist-test-slave-ffrd"
server_key: "a0294555913fdd35359e57810b75365c"
server_key_iv: "19af5602c334d4631ffc194b0b04d8de"
server_key_version: "encryptionkey@0"
I20260810 07:25:30.143483 23930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-1/wal/instance:
uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf"
format_stamp: "Formatted at 2026-08-10 07:25:30 on dist-test-slave-ffrd"
server_key: "a0294555913fdd35359e57810b75365c"
server_key_iv: "19af5602c334d4631ffc194b0b04d8de"
server_key_version: "encryptionkey@0"
I20260810 07:25:30.147400 23930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 07:25:30.150017 23953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:30.151396 23930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:25:30.151656 23930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-1/wal
uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf"
format_stamp: "Formatted at 2026-08-10 07:25:30 on dist-test-slave-ffrd"
server_key: "a0294555913fdd35359e57810b75365c"
server_key_iv: "19af5602c334d4631ffc194b0b04d8de"
server_key_version: "encryptionkey@0"
I20260810 07:25:30.151881 23930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:30.172721 23930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:30.176151 23930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:30.176383 23930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:30.183445 23930 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.253:33175
I20260810 07:25:30.183743 24039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.253:33175 every 8 connection(s)
I20260810 07:25:30.184541 23930 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-1/data/info.pb
I20260810 07:25:30.187567 24041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:30.191583 24041 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:25:30.193179 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 23930
Aug 10 07:25:30 dist-test-slave-ffrd krb5kdc[23110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346730, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.253@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM
Aug 10 07:25:30 dist-test-slave-ffrd krb5kdc[23110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.157.253@KRBTEST.COM
I20260810 07:25:30.206692 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.193672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52570 (local address 127.12.157.254:46027)
0810 07:25:30.193825 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:30.193829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:30.194885 (+ 1056us) server_negotiation.cc:407] Connection header received
0810 07:25:30.195728 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:30.195732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:30.195792 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:30.195873 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:30.197119 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.197736 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.198894 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.199120 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.199201 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.202060 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:30.202064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:30.202067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:30.202096 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:30.204084 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.204678 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.204681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.204684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.204782 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.205275 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.205278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.205279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.205621 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:30.205714 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:30.206350 (+ 636us) server_negotiation.cc:299] Negotiation successful
0810 07:25:30.206489 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 07:25:30.207683 24047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.194123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.0.0.1:52570)
0810 07:25:30.194751 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:30.194789 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:30.195431 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:30.196053 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:30.196059 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:30.196615 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:30.196969 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.196985 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.197887 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.197891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.198741 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.198750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.199266 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.199271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.199449 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.200231 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:30.200247 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:30.201866 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:30.204204 (+ 2338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.204213 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.204221 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.204571 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.205028 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.205031 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.205032 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.205166 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.205731 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:30.205738 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:30.206080 (+ 342us) client_negotiation.cc:769] Sending connection context
0810 07:25:30.206253 (+ 173us) client_negotiation.cc:240] Negotiation successful
0810 07:25:30.206608 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":52,"spinlock_wait_cycles":14464,"thread_start_us":195,"threads_started":1}
I20260810 07:25:30.207683 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.194122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56388 (local address 127.12.157.253:33175)
0810 07:25:30.194564 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:30.194581 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:30.194620 (+ 39us) server_negotiation.cc:407] Connection header received
0810 07:25:30.195379 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:30.195397 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:30.195590 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:30.195784 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:30.196624 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.197990 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.198852 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.199266 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.199454 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.202608 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:30.202617 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:30.202633 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:30.202664 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:30.204680 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.205253 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.205260 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.205263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.205346 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.205638 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.205640 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.205641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.206081 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:30.206282 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:30.206528 (+ 246us) server_negotiation.cc:299] Negotiation successful
0810 07:25:30.206653 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":299,"spinlock_wait_cycles":7040,"thread_start_us":135,"threads_started":1}
I20260810 07:25:30.209182 24041 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 07:25:30 dist-test-slave-ffrd krb5kdc[23110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346730, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.253@KRBTEST.COM for oryx/127.12.157.253@KRBTEST.COM
I20260810 07:25:30.217185 24047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.209714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:33175 (local address 127.0.0.1:56396)
0810 07:25:30.209810 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:30.209827 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:30.209895 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:30.210089 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:30.210092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:30.210261 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:30.210361 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.210366 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.211188 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.211191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.211774 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.211785 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.212327 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.212331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.212427 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.212925 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:30.212936 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:30.214196 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:30.215720 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.215729 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.215731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.215924 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.216210 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.216213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.216215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.216290 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.216650 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:30.216657 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:30.216751 (+ 94us) client_negotiation.cc:769] Sending connection context
0810 07:25:30.216854 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 07:25:30.217020 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260810 07:25:30.217190 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.209792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56396 (local address 127.12.157.253:33175)
0810 07:25:30.209905 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:30.209908 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:30.209923 (+ 15us) server_negotiation.cc:407] Connection header received
0810 07:25:30.209953 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:30.209955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:30.209985 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:30.210045 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:30.210470 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.211077 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.211911 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.212190 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.212302 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.214331 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:30.214336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:30.214344 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:30.214363 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:30.215608 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.216025 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.216033 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.216035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.216114 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.216383 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.216387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.216389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.216551 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:30.216662 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:30.216910 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 07:25:30.217075 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":43}
I20260810 07:25:30.220615 24041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf: Bootstrap starting.
I20260810 07:25:30.222950 24041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:30.223634 24041 log.cc:824] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:30.225493 24041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf: No bootstrap required, opened a new log
I20260810 07:25:30.228353 24041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } }
I20260810 07:25:30.228540 24041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:30.228572 24041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f2fdbbd52b1447a8119c5d0ba547dbf, State: Initialized, Role: FOLLOWER
I20260810 07:25:30.228925 24041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } }
I20260810 07:25:30.229463 24049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } } }
I20260810 07:25:30.229579 24049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:25:30.230111 24041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [sys.catalog]: configured and running, proceeding with master startup.
W20260810 07:25:30.235001 24068 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:25:30.235095 24068 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:25:30.237013 23930 master_runner.cc:428] Detected that this master 7f2fdbbd52b1447a8119c5d0ba547dbf is joining an existing cluster
I20260810 07:25:30.237078 23930 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.157.253:33175
I20260810 07:25:30.248226 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.239597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56410 (local address 127.12.157.253:33175)
0810 07:25:30.239698 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:30.239702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:30.240033 (+ 331us) server_negotiation.cc:407] Connection header received
0810 07:25:30.240139 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:30.240145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:30.240194 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:30.240267 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:30.240997 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.241784 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.242729 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.242952 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.243064 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.244392 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:30.244396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:30.244399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:30.244427 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:30.246367 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.246831 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.246836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.246838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.246921 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.247226 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.247229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.247231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.247468 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:30.247561 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:30.247910 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 07:25:30.248026 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":31}
I20260810 07:25:30.248898 24077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.239614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:33175 (local address 127.0.0.1:56410)
0810 07:25:30.239929 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:30.239944 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:30.240052 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:30.240375 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:30.240377 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:30.240701 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:30.240859 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.240864 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.241935 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.241938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.242557 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.242565 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.243075 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.243079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.243190 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.243729 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:30.243777 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:30.244242 (+ 465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:30.246470 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.246474 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.246476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.246730 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.247018 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.247023 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.247025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.247120 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.247622 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:30.247624 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:30.247729 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 07:25:30.248579 (+ 850us) client_negotiation.cc:240] Negotiation successful
0810 07:25:30.248710 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":4,"thread_start_us":126,"threads_started":1}
I20260810 07:25:30.250192 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.239540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52574 (local address 127.12.157.254:46027)
0810 07:25:30.239682 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:30.239689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:30.240176 (+ 487us) server_negotiation.cc:407] Connection header received
0810 07:25:30.240294 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:30.240297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:30.240355 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:30.240436 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:30.241931 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.242735 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.243951 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.244303 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.244400 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.246381 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:30.246385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:30.246388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:30.246414 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:30.248294 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.248783 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.248787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.248789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.248885 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.249240 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.249251 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.249253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.249471 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:30.249581 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:30.249906 (+ 325us) server_negotiation.cc:299] Negotiation successful
0810 07:25:30.250031 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 07:25:30.250640 24078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.239720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.0.0.1:52574)
0810 07:25:30.240088 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:30.240101 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:30.240195 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:30.240467 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:30.240470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:30.240725 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:30.240850 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.240858 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.242878 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.242882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.243815 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.243826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.244512 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.244514 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.244596 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.245168 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:30.245181 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:30.246233 (+ 1052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:30.248430 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.248433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.248435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.248672 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.249031 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.249035 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.249037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.249128 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.249625 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:30.249628 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:30.249714 (+ 86us) client_negotiation.cc:769] Sending connection context
0810 07:25:30.250349 (+ 635us) client_negotiation.cc:240] Negotiation successful
0810 07:25:30.250494 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":97,"threads_started":1}
Aug 10 07:25:30 dist-test-slave-ffrd krb5kdc[23110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346729, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.254@KRBTEST.COM for oryx/127.12.157.253@KRBTEST.COM
I20260810 07:25:30.264144 24045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.254807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56424 (local address 127.12.157.253:33175)
0810 07:25:30.254918 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:30.254924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:30.255238 (+ 314us) server_negotiation.cc:407] Connection header received
0810 07:25:30.255449 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:30.255452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:30.255502 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:30.255595 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:30.256316 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.256809 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.257793 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.258025 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.258138 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.260374 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:30.260378 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:30.260381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:30.260404 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:30.262254 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.262861 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.262865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.262867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.262950 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.263240 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.263243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.263245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.263426 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:30.263539 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:30.263882 (+ 343us) server_negotiation.cc:299] Negotiation successful
0810 07:25:30.263997 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 07:25:30.264170 24079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.254741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:33175 (local address 127.0.0.1:56424)
0810 07:25:30.255118 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:30.255159 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:30.255315 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:30.255653 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:30.255661 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:30.256059 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:30.256192 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.256199 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.256923 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.256928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.257580 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.257590 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.258142 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.258145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.258264 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.258823 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:30.258841 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:30.260230 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:30.262404 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.262417 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.262425 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.262744 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.263049 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.263052 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.263054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.263144 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.263539 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:30.263547 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:30.263670 (+ 123us) client_negotiation.cc:769] Sending connection context
0810 07:25:30.263823 (+ 153us) client_negotiation.cc:240] Negotiation successful
0810 07:25:30.264009 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":115,"threads_started":1}
I20260810 07:25:30.265751 23186 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.157.253:33175: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf"
member_type: NON_VOTER
last_known_addr {
host: "127.12.157.253"
port: 33175
}
attrs {
promote: true
}
}
dest_uuid: "eebcdb40543d45c5aef6817f84cef282"
cas_config_opid_index: -1
I20260810 07:25:30.266372 23186 consensus_queue.cc:237] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: NON_VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: true } }
I20260810 07:25:30.268245 24085 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 7f2fdbbd52b1447a8119c5d0ba547dbf (127.12.157.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: NON_VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: true } } }
I20260810 07:25:30.268908 23998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:30.270038 24085 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.157.253:33175
I20260810 07:25:30.270391 23998 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [term 1 FOLLOWER]: Refusing update from remote peer eebcdb40543d45c5aef6817f84cef282: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 07:25:30.270514 23930 master.cc:537] Master@127.12.157.253:33175 shutting down...
I20260810 07:25:30.270664 24085 sys_catalog.cc:455] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "eebcdb40543d45c5aef6817f84cef282" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: NON_VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: true } } }
I20260810 07:25:30.270776 24085 sys_catalog.cc:458] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:30.271276 24086 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: NON_VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 07:25:30.273343 23168 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260810 07:25:30.275722 23930 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:25:30.276021 23930 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:30.276147 23930 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf: stopping tablet replica
WARNING: no policy specified for oryx/127.12.157.252@KRBTEST.COM; defaulting to no policy
I20260810 07:25:30.280653 24079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.280301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:33175 (local address 127.0.0.1:56428)
0810 07:25:30.280436 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:30.280513 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.253:33175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 07:25:30.280884 23168 consensus_peers.cc:487] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 -> Peer 7f2fdbbd52b1447a8119c5d0ba547dbf (127.12.157.253:33175): Couldn't send request to peer 7f2fdbbd52b1447a8119c5d0ba547dbf. Status: Network error: Client connection negotiation failed: client connection to 127.12.157.253:33175: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.157.252@KRBTEST.COM" created.
I20260810 07:25:30.291514 23930 master.cc:552] Master@127.12.157.253:33175 shutdown complete.
I20260810 07:25:30.291575 23930 master_runner.cc:305] Clearing existing system tablet
I20260810 07:25:30.291836 23930 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:30.291896 23930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:30.293546 23930 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 07:25:30.295060 24099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:30.295277 23930 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:30.295436 23930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 07:25:30.295482 23930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-1/wal
uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf"
format_stamp: "Formatted at 2026-08-10 07:25:30 on dist-test-slave-ffrd"
server_key: "a0294555913fdd35359e57810b75365c"
server_key_iv: "19af5602c334d4631ffc194b0b04d8de"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.157.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.252.keytab.
Entry for principal oryx/127.12.157.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.252.keytab.
I20260810 07:25:30.297698 23930 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:30.298396 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.252:39867
--webserver_interface=127.12.157.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.252.keytab
--principal=oryx/127.12.157.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:43127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:46027,127.12.157.253:33175,127.12.157.252:39867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:25:30.300736 23930 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 07:25:30.300861 23930 log.cc:1196] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 07:25:30.301213 23930 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf: Deleting consensus metadata
I20260810 07:25:30.301405 23930 master_runner.cc:315] Copying system tablet from 127.12.157.254:46027
I20260810 07:25:30.302758 23930 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf: tablet copy: Beginning tablet copy session from remote peer at address 127.12.157.254:46027
I20260810 07:25:30.310889 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.303442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52576 (local address 127.12.157.254:46027)
0810 07:25:30.303580 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:30.303586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:30.303818 (+ 232us) server_negotiation.cc:407] Connection header received
0810 07:25:30.303978 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:30.303981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:30.304034 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:30.304123 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:30.304599 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.305255 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.306105 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.306331 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.306436 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.307640 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:30.307643 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:30.307651 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:30.307676 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:30.309199 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.309627 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.309631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.309633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.309716 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.310007 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.310011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.310013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.310196 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:30.310308 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:30.310583 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 07:25:30.310707 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 07:25:30.310935 24107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.303361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.0.0.1:52576)
0810 07:25:30.303716 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:30.303729 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:30.303871 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:30.304141 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:30.304144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:30.304364 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:30.304472 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.304478 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.305376 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.305380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.305968 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.305977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.306455 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.306459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.306555 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.307120 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:30.307133 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:30.307487 (+ 354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:30.309313 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.309318 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.309320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.309528 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.309823 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.309826 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.309828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.309917 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.310309 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:30.310313 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:30.310416 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 07:25:30.310522 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 07:25:30.310730 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":102,"threads_started":1}
I20260810 07:25:30.311790 23227 tablet_copy_service.cc:140] P eebcdb40543d45c5aef6817f84cef282: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 7f2fdbbd52b1447a8119c5d0ba547dbf ({username='oryx', principal='oryx/127.12.157.253@KRBTEST.COM'} at 127.0.0.1:52576)
I20260810 07:25:30.311980 23227 tablet_copy_service.cc:161] P eebcdb40543d45c5aef6817f84cef282: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 7f2fdbbd52b1447a8119c5d0ba547dbf at {username='oryx', principal='oryx/127.12.157.253@KRBTEST.COM'} at 127.0.0.1:52576: session id = 7f2fdbbd52b1447a8119c5d0ba547dbf-00000000000000000000000000000000
I20260810 07:25:30.313586 23227 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 07:25:30.315163 23930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 07:25:30.317266 23930 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf: tablet copy: Starting download of 0 data blocks...
I20260810 07:25:30.317488 23930 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf: tablet copy: Starting download of 1 WAL segments...
I20260810 07:25:30.319165 23930 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 07:25:30.320998 23227 tablet_copy_service.cc:342] P eebcdb40543d45c5aef6817f84cef282: Request end of tablet copy session 7f2fdbbd52b1447a8119c5d0ba547dbf-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.157.253@KRBTEST.COM'} at 127.0.0.1:52576
I20260810 07:25:30.321137 23227 tablet_copy_service.cc:433] P eebcdb40543d45c5aef6817f84cef282: ending tablet copy session 7f2fdbbd52b1447a8119c5d0ba547dbf-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 7f2fdbbd52b1447a8119c5d0ba547dbf
I20260810 07:25:30.328501 23930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:30.330394 24108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:30.330390 24113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:30.330681 24109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:30.333164 23930 server_base.cc:1034] running on GCE node
I20260810 07:25:30.333357 23930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:30.333691 23930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:30.334841 23930 hybrid_clock.cc:648] HybridClock initialized: now 1786346730334826 us; error 34 us; skew 500 ppm
Aug 10 07:25:30 dist-test-slave-ffrd krb5kdc[23110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346730, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:30.336367 23930 init.cc:377] Logged in from keytab as oryx/127.12.157.253@KRBTEST.COM (short username oryx)
I20260810 07:25:30.336886 23930 webserver.cc:460] Webserver started at http://127.12.157.253:41423/ using document root <none> and password file <none>
I20260810 07:25:30.337143 23930 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:30.337198 23930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:30.338490 23930 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:25:30.339488 24121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:30.339723 23930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 07:25:30.339821 23930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-1/wal
uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf"
format_stamp: "Formatted at 2026-08-10 07:25:30 on dist-test-slave-ffrd"
server_key: "a0294555913fdd35359e57810b75365c"
server_key_iv: "19af5602c334d4631ffc194b0b04d8de"
server_key_version: "encryptionkey@0"
I20260810 07:25:30.339926 23930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:30.345534 23930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:30.347478 23930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:30.353251 23930 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.253:33175
I20260810 07:25:30.353890 23930 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-1/data/info.pb
I20260810 07:25:30.354316 24210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.253:33175 every 8 connection(s)
I20260810 07:25:30.355576 24212 sys_catalog.cc:263] Verifying existing consensus state
I20260810 07:25:30.356420 24212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf: Bootstrap starting.
I20260810 07:25:30.365108 24212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 07:25:30.365427 24212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf: Bootstrap complete.
I20260810 07:25:30.365936 24212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: NON_VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: true } }
I20260810 07:25:30.366077 24212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7f2fdbbd52b1447a8119c5d0ba547dbf, State: Initialized, Role: LEARNER
I20260810 07:25:30.366223 24212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: NON_VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: true } }
I20260810 07:25:30.366582 24216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: NON_VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: true } } }
I20260810 07:25:30.366741 24216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [sys.catalog]: This master's current role is: LEARNER
I20260810 07:25:30.366750 24212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:25:30.369771 24237 catalog_manager.cc:1259] Loaded cluster ID: 45c8a83eed7741dfa445f32c118347c1
I20260810 07:25:30.369884 24237 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf: loading cluster ID for follower catalog manager: success
I20260810 07:25:30.371817 24237 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf: acquiring CA information for follower catalog manager: success
I20260810 07:25:30.373260 24237 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 07:25:30.469789 24101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:30.470032 24101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:30.470073 24101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:30.474197 24101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:25:30.474267 24101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:30.474288 24101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:30.474305 24101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:25:30.474339 24101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:25:30.478857 23488 heartbeater.cc:503] Master 127.12.157.254:46027 was elected leader, sending a full tablet report...
I20260810 07:25:30.480149 24101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:43127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:46027,127.12.157.253:33175,127.12.157.252:39867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.252.keytab
--principal=oryx/127.12.157.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.252:39867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.157.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:30.481408 24101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:30.482488 24101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:30.491171 24246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:30.491425 24101 server_base.cc:1034] running on GCE node
W20260810 07:25:30.491166 24242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:30.491191 24244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:30.492262 24101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:30.493464 24101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:30.494659 24101 hybrid_clock.cc:648] HybridClock initialized: now 1786346730494629 us; error 42 us; skew 500 ppm
Aug 10 07:25:30 dist-test-slave-ffrd krb5kdc[23110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346730, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:30.497866 24101 init.cc:377] Logged in from keytab as oryx/127.12.157.252@KRBTEST.COM (short username oryx)
I20260810 07:25:30.498780 24101 webserver.cc:460] Webserver started at http://127.12.157.252:33065/ using document root <none> and password file <none>
I20260810 07:25:30.499375 24101 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:30.499475 24101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:30.499724 24101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:30.501883 24101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-2/data/instance:
uuid: "1115ade1c930472ea39553bfd9614863"
format_stamp: "Formatted at 2026-08-10 07:25:30 on dist-test-slave-ffrd"
server_key: "08f8c8bfe94131710fe9ecd0927a90ab"
server_key_iv: "81fbc06280a9c1d2170e3cbaeb3ed440"
server_key_version: "encryptionkey@0"
I20260810 07:25:30.502468 24101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-2/wal/instance:
uuid: "1115ade1c930472ea39553bfd9614863"
format_stamp: "Formatted at 2026-08-10 07:25:30 on dist-test-slave-ffrd"
server_key: "08f8c8bfe94131710fe9ecd0927a90ab"
server_key_iv: "81fbc06280a9c1d2170e3cbaeb3ed440"
server_key_version: "encryptionkey@0"
I20260810 07:25:30.506346 24101 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 07:25:30.508857 24256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:30.510150 24101 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 07:25:30.510294 24101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-2/wal
uuid: "1115ade1c930472ea39553bfd9614863"
format_stamp: "Formatted at 2026-08-10 07:25:30 on dist-test-slave-ffrd"
server_key: "08f8c8bfe94131710fe9ecd0927a90ab"
server_key_iv: "81fbc06280a9c1d2170e3cbaeb3ed440"
server_key_version: "encryptionkey@0"
I20260810 07:25:30.510406 24101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:30.529682 24101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:30.532912 24101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:30.533175 24101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:30.541095 24101 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.252:39867
I20260810 07:25:30.541452 24338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.252:39867 every 8 connection(s)
I20260810 07:25:30.542222 24101 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-2/data/info.pb
I20260810 07:25:30.545070 24339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:30.547979 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 24101
I20260810 07:25:30.549321 24339 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 07:25:30 dist-test-slave-ffrd krb5kdc[23110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346730, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.252@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM
Aug 10 07:25:30 dist-test-slave-ffrd krb5kdc[23110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.157.252@KRBTEST.COM
I20260810 07:25:30.562399 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.551398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52584 (local address 127.12.157.254:46027)
0810 07:25:30.551529 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:30.551533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:30.551784 (+ 251us) server_negotiation.cc:407] Connection header received
0810 07:25:30.551997 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:30.552001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:30.552051 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:30.552107 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:30.553217 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.553754 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.554929 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.555195 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.555332 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.558027 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:30.558034 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:30.558039 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:30.558073 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:30.559979 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.560640 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.560646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.560651 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.560756 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.561176 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.561179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.561181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.561440 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:30.561541 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:30.562095 (+ 554us) server_negotiation.cc:299] Negotiation successful
0810 07:25:30.562230 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 07:25:30.563221 24345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.551059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.0.0.1:52584)
0810 07:25:30.551662 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:30.551711 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:30.551868 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:30.552282 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:30.552286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:30.552700 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:30.553042 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.553058 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.553943 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.553947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.554782 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.554792 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.555520 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.555527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.555665 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.556208 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:30.556224 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:30.557848 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:30.560116 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.560125 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.560134 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.560530 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.560857 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.560861 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.560863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.561075 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.561553 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:30.561561 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:30.561817 (+ 256us) client_negotiation.cc:769] Sending connection context
0810 07:25:30.562006 (+ 189us) client_negotiation.cc:240] Negotiation successful
0810 07:25:30.562371 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":57,"thread_start_us":331,"threads_started":1}
I20260810 07:25:30.563228 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.549762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32850 (local address 127.12.157.252:39867)
0810 07:25:30.550201 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:30.550217 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:30.550254 (+ 37us) server_negotiation.cc:407] Connection header received
0810 07:25:30.550906 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:30.550923 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:30.551332 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:30.551692 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:30.552549 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.553964 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.554779 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.555198 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.555368 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.558661 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:30.558671 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:30.558685 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:30.558713 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:30.560565 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.561175 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.561181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.561183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.561271 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.561578 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.561581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.561583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.561884 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:30.562101 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:30.562301 (+ 200us) server_negotiation.cc:299] Negotiation successful
0810 07:25:30.562452 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":298,"thread_start_us":133,"threads_started":1}
I20260810 07:25:30.564450 24339 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 07:25:30 dist-test-slave-ffrd krb5kdc[23110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346730, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.252@KRBTEST.COM for oryx/127.12.157.253@KRBTEST.COM
I20260810 07:25:30.574246 24345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.565026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:33175 (local address 127.0.0.1:56444)
0810 07:25:30.565440 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:30.565457 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:30.565536 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:30.565830 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:30.565833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:30.565997 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:30.566089 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.566094 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.567094 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.567099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.567700 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.567738 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.568299 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.568303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.568411 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.568973 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:30.568992 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:30.570444 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:30.572647 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.572650 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.572652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.572860 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.573230 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.573232 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.573234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.573317 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.573793 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:30.573796 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:30.573888 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 07:25:30.573992 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 07:25:30.574120 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330}
I20260810 07:25:30.574330 24347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.565158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56444 (local address 127.12.157.253:33175)
0810 07:25:30.565574 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:30.565581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:30.565597 (+ 16us) server_negotiation.cc:407] Connection header received
0810 07:25:30.565648 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:30.565650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:30.565688 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:30.565817 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:30.566210 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.566985 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.567861 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.568176 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.568299 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.570623 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:30.570627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:30.570629 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:30.570653 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:30.572539 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.573009 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.573012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.573013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.573122 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.573451 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.573454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.573455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.573668 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:30.573757 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:30.574053 (+ 296us) server_negotiation.cc:299] Negotiation successful
0810 07:25:30.574167 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":131,"threads_started":1}
I20260810 07:25:30.575163 24339 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.252" port: 39867 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 07:25:30 dist-test-slave-ffrd krb5kdc[23110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346730, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.252@KRBTEST.COM for oryx/127.12.157.252@KRBTEST.COM
I20260810 07:25:30.583063 24345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.575661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:39867 (local address 127.0.0.1:32864)
0810 07:25:30.575781 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:30.575795 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:30.575895 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:30.576117 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:30.576119 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:30.576248 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:30.576320 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.576324 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.577162 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.577165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.577668 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.577677 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.578125 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.578128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.578187 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.578623 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:30.578634 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:30.579879 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:30.581613 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.581617 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.581619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.581811 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.582097 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.582100 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.582102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.582182 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.582547 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:30.582550 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:30.582632 (+ 82us) client_negotiation.cc:769] Sending connection context
0810 07:25:30.582751 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 07:25:30.582890 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 07:25:30.583065 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.575752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32864 (local address 127.12.157.252:39867)
0810 07:25:30.575881 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:30.575885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:30.575897 (+ 12us) server_negotiation.cc:407] Connection header received
0810 07:25:30.575988 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:30.575990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:30.576025 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:30.576089 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:30.576462 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.577045 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.577775 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.578022 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.578104 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.580026 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:30.580029 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:30.580030 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:30.580050 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:30.581486 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.581907 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.581913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.581915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.581998 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.582271 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.582276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.582277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.582447 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:30.582558 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:30.582798 (+ 240us) server_negotiation.cc:299] Negotiation successful
0810 07:25:30.582945 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":56}
I20260810 07:25:30.586766 24339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863: Bootstrap starting.
I20260810 07:25:30.589540 24339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:30.590268 24339 log.cc:824] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:30.592242 24339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863: No bootstrap required, opened a new log
I20260810 07:25:30.595602 24339 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 39867 } }
I20260810 07:25:30.595800 24339 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:30.595834 24339 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1115ade1c930472ea39553bfd9614863, State: Initialized, Role: FOLLOWER
I20260810 07:25:30.596206 24339 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 39867 } }
I20260810 07:25:30.596680 24350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 39867 } } }
I20260810 07:25:30.596839 24350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:25:30.597469 24339 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 07:25:30.602572 24370 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:25:30.602662 24370 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:25:30.606173 24101 master_runner.cc:428] Detected that this master 1115ade1c930472ea39553bfd9614863 is joining an existing cluster
I20260810 07:25:30.606245 24101 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.157.252:39867
I20260810 07:25:30.617240 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 23287
I20260810 07:25:30.618120 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.608422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52588 (local address 127.12.157.254:46027)
0810 07:25:30.608559 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:30.608565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:30.609100 (+ 535us) server_negotiation.cc:407] Connection header received
0810 07:25:30.609221 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:30.609224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:30.609266 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:30.609361 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:30.609989 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.610804 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.612165 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.612484 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.612559 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.614285 (+ 1726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:30.614289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:30.614292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:30.614319 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:30.616194 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.616741 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.616745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.616747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.616834 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.617171 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.617175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.617176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.617417 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:30.617490 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:30.617839 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 07:25:30.617959 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 07:25:30.618588 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.608360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.0.0.1:52588)
0810 07:25:30.609007 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:30.609022 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:30.609129 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:30.609394 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:30.609398 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:30.609669 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:30.609845 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.609852 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.610952 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.610955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.612001 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.612015 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.612725 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.612744 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.613005 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.613585 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:30.613600 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:30.614135 (+ 535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:30.616337 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.616341 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.616343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.616604 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.616972 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.616975 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.616977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.617065 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.617565 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:30.617568 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:30.617655 (+ 87us) client_negotiation.cc:769] Sending connection context
0810 07:25:30.618264 (+ 609us) client_negotiation.cc:240] Negotiation successful
0810 07:25:30.618417 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":75,"thread_start_us":99,"threads_started":1}
I20260810 07:25:30.619982 24347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.608709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56446 (local address 127.12.157.253:33175)
0810 07:25:30.608833 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:30.608837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:30.609418 (+ 581us) server_negotiation.cc:407] Connection header received
0810 07:25:30.609585 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:30.609590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:30.609645 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:30.609726 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:30.610329 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.611094 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.613553 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.613968 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.614058 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.615398 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:30.615403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:30.615406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:30.615438 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:30.617214 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.617737 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.617742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.617743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.617838 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.618385 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.618391 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.618396 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.618674 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:30.618963 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:30.619585 (+ 622us) server_negotiation.cc:299] Negotiation successful
0810 07:25:30.619748 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 07:25:30.621817 24381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.608423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:33175 (local address 127.0.0.1:56446)
0810 07:25:30.609340 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:30.609353 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:30.609479 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:30.609845 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:30.609848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:30.610058 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:30.610173 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.610179 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.611282 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.611285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.613070 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.613082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.614258 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.614261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.614348 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.614782 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:30.614801 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:30.615159 (+ 358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:30.617350 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.617356 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.617361 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.617622 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.617978 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.617984 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.617989 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.618117 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.619060 (+ 943us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:30.619065 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:30.619200 (+ 135us) client_negotiation.cc:769] Sending connection context
0810 07:25:30.620196 (+ 996us) client_negotiation.cc:240] Negotiation successful
0810 07:25:30.620689 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"mutex_wait_us":69,"thread_start_us":108,"threads_started":1}
Aug 10 07:25:30 dist-test-slave-ffrd krb5kdc[23110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346729, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.254@KRBTEST.COM for oryx/127.12.157.252@KRBTEST.COM
I20260810 07:25:30.637915 24383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.608819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:39867 (local address 127.0.0.1:32876)
0810 07:25:30.609700 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:30.609712 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:30.609830 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:30.610128 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:30.610133 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:30.610354 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:30.610470 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.610479 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.611449 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.611452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.612414 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.612426 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.613282 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.613285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.613384 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.633700 (+ 20316us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:30.633717 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:30.634244 (+ 527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:30.636268 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.636271 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.636273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.636516 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.636771 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.636774 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.636776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.636845 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.637293 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:30.637295 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:30.637367 (+ 72us) client_negotiation.cc:769] Sending connection context
0810 07:25:30.637474 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 07:25:30.637619 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260810 07:25:30.638375 24341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.608824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32876 (local address 127.12.157.252:39867)
0810 07:25:30.608936 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:30.608968 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:30.609773 (+ 805us) server_negotiation.cc:407] Connection header received
0810 07:25:30.609923 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:30.609926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:30.609963 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:30.610009 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:30.610588 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.611316 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.612538 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.613080 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.613182 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.634404 (+ 21222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:30.634408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:30.634411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:30.634445 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:30.636151 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.636616 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.636619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.636621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.636689 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.636924 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.636928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.636930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.637172 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:30.637261 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:30.638128 (+ 867us) server_negotiation.cc:299] Negotiation successful
0810 07:25:30.638239 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":34}
I20260810 07:25:30.641355 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:36441
--local_ip_for_outbound_sockets=127.12.157.193
--tserver_master_addrs=127.12.157.254:46027
--webserver_port=46827
--webserver_interface=127.12.157.193
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.193.keytab
--principal=oryx/127.12.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:43127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.157.254:46027,127.12.157.253:33175
--tserver_master_addrs=127.12.157.254:46027,127.12.157.253:33175,127.12.157.252:39867 with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:25:30.642642 24079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.622536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:39867 (local address 127.0.0.1:32882)
0810 07:25:30.622688 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:30.622702 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:30.622782 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:30.623648 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:30.623654 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:30.623909 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:30.624064 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.624078 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.625205 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.625209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.626083 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.626094 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.626736 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.626743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.626889 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.627471 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:30.627494 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:30.629377 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:30.640691 (+ 11314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.640695 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.640697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.641067 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.641429 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.641433 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.641435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.641549 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.642073 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:30.642076 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:30.642193 (+ 117us) client_negotiation.cc:769] Sending connection context
0810 07:25:30.642312 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 07:25:30.642479 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 07:25:30.642894 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.622809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32882 (local address 127.12.157.252:39867)
0810 07:25:30.623327 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:30.623335 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:30.623348 (+ 13us) server_negotiation.cc:407] Connection header received
0810 07:25:30.623428 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:30.623431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:30.623491 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:30.623585 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:30.624209 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.625072 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.626233 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.626599 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.626697 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.629608 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:30.629616 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:30.629623 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:30.629661 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:30.640546 (+ 10885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.641198 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.641201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.641203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.641285 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.641673 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.641679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.641683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.641953 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:30.642064 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:30.642397 (+ 333us) server_negotiation.cc:299] Negotiation successful
0810 07:25:30.642634 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":350,"threads_started":1}
I20260810 07:25:30.644012 23186 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.157.252:39867: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1115ade1c930472ea39553bfd9614863"
member_type: NON_VOTER
last_known_addr {
host: "127.12.157.252"
port: 39867
}
attrs {
promote: true
}
}
dest_uuid: "eebcdb40543d45c5aef6817f84cef282"
cas_config_opid_index: 5
I20260810 07:25:30.644656 23186 consensus_queue.cc:237] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: NON_VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: true } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: NON_VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: true } }
I20260810 07:25:30.646276 24085 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1115ade1c930472ea39553bfd9614863 (127.12.157.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: NON_VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: true } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: NON_VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: true } } }
I20260810 07:25:30.646416 24299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:30.647732 24085 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.157.252:39867
I20260810 07:25:30.647902 24085 sys_catalog.cc:455] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "eebcdb40543d45c5aef6817f84cef282" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: NON_VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: true } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: NON_VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: true } } }
I20260810 07:25:30.647990 24085 sys_catalog.cc:458] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:30.648180 24299 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [term 1 FOLLOWER]: Refusing update from remote peer eebcdb40543d45c5aef6817f84cef282: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 07:25:30.648840 24101 master.cc:537] Master@127.12.157.252:39867 shutting down...
I20260810 07:25:30.649017 24086 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: NON_VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 07:25:30.651914 23168 consensus_peers.cc:487] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 -> Peer 1115ade1c930472ea39553bfd9614863 (127.12.157.252:39867): Couldn't send request to peer 1115ade1c930472ea39553bfd9614863. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 07:25:30.655977 24079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.645949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:33175 (local address 127.0.0.1:56454)
0810 07:25:30.648425 (+ 2476us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:30.648439 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:30.648523 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:30.651782 (+ 3259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:30.651786 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:30.651804 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:30.651892 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.651898 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.653084 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.653087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.654352 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.654364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.655532 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.655536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.655625 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.655675 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 07:25:30.655749 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 07:25:30.655825 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2401}
I20260810 07:25:30.656348 24347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.646171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56454 (local address 127.12.157.253:33175)
0810 07:25:30.648329 (+ 2158us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:30.648334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:30.648605 (+ 271us) server_negotiation.cc:407] Connection header received
0810 07:25:30.651464 (+ 2859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:30.651468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:30.651541 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:30.651645 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:30.652022 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.652866 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.654512 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.655189 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.655306 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.655469 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:30.656139 (+ 670us) server_negotiation.cc:299] Negotiation successful
0810 07:25:30.656223 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2066}
I20260810 07:25:30.656802 24162 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [term 1 LEARNER]: Refusing update from remote peer eebcdb40543d45c5aef6817f84cef282: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 07:25:30.657222 24086 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: NON_VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 07:25:30.659390 24162 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1115ade1c930472ea39553bfd9614863 (127.12.157.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: NON_VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: true } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: NON_VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: true } } }
I20260810 07:25:30.661535 24216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader eebcdb40543d45c5aef6817f84cef282. Latest consensus state: current_term: 1 leader_uuid: "eebcdb40543d45c5aef6817f84cef282" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: NON_VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: true } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: NON_VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: true } } }
I20260810 07:25:30.661664 24216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [sys.catalog]: This master's current role is: LEARNER
I20260810 07:25:30.661752 24216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eebcdb40543d45c5aef6817f84cef282" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: NON_VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: true } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: NON_VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: true } } }
I20260810 07:25:30.661829 24216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [sys.catalog]: This master's current role is: LEARNER
I20260810 07:25:30.662418 24086 raft_consensus.cc:1050] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282: attempting to promote NON_VOTER 7f2fdbbd52b1447a8119c5d0ba547dbf to VOTER
I20260810 07:25:30.662899 24086 consensus_queue.cc:237] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: NON_VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: true } }
I20260810 07:25:30.665349 24162 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [term 1 LEARNER]: Refusing update from remote peer eebcdb40543d45c5aef6817f84cef282: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 07:25:30.665722 24085 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260810 07:25:30.666059 23168 consensus_peers.cc:487] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 -> Peer 1115ade1c930472ea39553bfd9614863 (127.12.157.252:39867): Couldn't send request to peer 1115ade1c930472ea39553bfd9614863. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 07:25:30.666597 24101 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:25:30.666903 24101 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:30.667017 24101 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863: stopping tablet replica
I20260810 07:25:30.668202 24085 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7f2fdbbd52b1447a8119c5d0ba547dbf (127.12.157.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: NON_VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: true } } }
I20260810 07:25:30.668687 24162 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 7f2fdbbd52b1447a8119c5d0ba547dbf (127.12.157.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: NON_VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: true } } }
I20260810 07:25:30.669858 24085 sys_catalog.cc:455] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "eebcdb40543d45c5aef6817f84cef282" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: NON_VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: true } } }
I20260810 07:25:30.669970 24085 sys_catalog.cc:458] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:30.670465 24086 sys_catalog.cc:455] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eebcdb40543d45c5aef6817f84cef282" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: NON_VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: true } } }
I20260810 07:25:30.670619 24086 sys_catalog.cc:458] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:30.671603 24388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eebcdb40543d45c5aef6817f84cef282" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: NON_VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: true } } }
I20260810 07:25:30.671720 24388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:25:30.683316 24101 master.cc:552] Master@127.12.157.252:39867 shutdown complete.
I20260810 07:25:30.683387 24101 master_runner.cc:305] Clearing existing system tablet
I20260810 07:25:30.683642 24101 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:30.683712 24101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:30.685189 24101 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 07:25:30.686283 24398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:30.686591 24101 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:30.686728 24101 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:30.686764 24101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-2/wal
uuid: "1115ade1c930472ea39553bfd9614863"
format_stamp: "Formatted at 2026-08-10 07:25:30 on dist-test-slave-ffrd"
server_key: "08f8c8bfe94131710fe9ecd0927a90ab"
server_key_iv: "81fbc06280a9c1d2170e3cbaeb3ed440"
server_key_version: "encryptionkey@0"
I20260810 07:25:30.689291 24101 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:30.691802 24101 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 07:25:30.691891 24101 log.cc:1196] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 07:25:30.692140 24101 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863: Deleting consensus metadata
I20260810 07:25:30.692308 24101 master_runner.cc:315] Copying system tablet from 127.12.157.254:46027
I20260810 07:25:30.693636 24101 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863: tablet copy: Beginning tablet copy session from remote peer at address 127.12.157.254:46027
I20260810 07:25:30.704871 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.695616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52590 (local address 127.12.157.254:46027)
0810 07:25:30.695777 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:30.695782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:30.695803 (+ 21us) server_negotiation.cc:407] Connection header received
0810 07:25:30.695987 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:30.695991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:30.696043 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:30.696096 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:30.696783 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.697690 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.699064 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.699453 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.699582 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.701124 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:30.701128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:30.701131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:30.701158 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:30.702792 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.703300 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.703304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.703306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.703394 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.703723 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.703727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.703729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.704071 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:30.704156 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:30.704549 (+ 393us) server_negotiation.cc:299] Negotiation successful
0810 07:25:30.704673 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 07:25:30.705742 24407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.695390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.0.0.1:52590)
0810 07:25:30.695728 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:30.695743 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:30.695850 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:30.696254 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:30.696257 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:30.696505 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:30.696636 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.696643 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.697816 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.697820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.698923 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.698935 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.699769 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.699773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.699870 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.700462 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:30.700485 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:30.700931 (+ 446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:30.702923 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.702927 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.702929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.703182 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.703523 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.703526 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.703528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.703617 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.704238 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:30.704250 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:30.704356 (+ 106us) client_negotiation.cc:769] Sending connection context
0810 07:25:30.705393 (+ 1037us) client_negotiation.cc:240] Negotiation successful
0810 07:25:30.705551 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":99,"threads_started":1}
I20260810 07:25:30.706467 23227 tablet_copy_service.cc:140] P eebcdb40543d45c5aef6817f84cef282: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1115ade1c930472ea39553bfd9614863 ({username='oryx', principal='oryx/127.12.157.252@KRBTEST.COM'} at 127.0.0.1:52590)
I20260810 07:25:30.706566 23227 tablet_copy_service.cc:161] P eebcdb40543d45c5aef6817f84cef282: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1115ade1c930472ea39553bfd9614863 at {username='oryx', principal='oryx/127.12.157.252@KRBTEST.COM'} at 127.0.0.1:52590: session id = 1115ade1c930472ea39553bfd9614863-00000000000000000000000000000000
I20260810 07:25:30.707648 23227 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 07:25:30.709650 24101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 07:25:30.712258 24101 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863: tablet copy: Starting download of 0 data blocks...
I20260810 07:25:30.712450 24101 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863: tablet copy: Starting download of 1 WAL segments...
I20260810 07:25:30.714148 24101 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 07:25:30.715598 23700 heartbeater.cc:503] Master 127.12.157.254:46027 was elected leader, sending a full tablet report...
I20260810 07:25:30.716298 23227 tablet_copy_service.cc:342] P eebcdb40543d45c5aef6817f84cef282: Request end of tablet copy session 1115ade1c930472ea39553bfd9614863-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.157.252@KRBTEST.COM'} at 127.0.0.1:52590
I20260810 07:25:30.716378 23227 tablet_copy_service.cc:433] P eebcdb40543d45c5aef6817f84cef282: ending tablet copy session 1115ade1c930472ea39553bfd9614863-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1115ade1c930472ea39553bfd9614863
I20260810 07:25:30.724126 24101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:30.725855 24409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:30.725872 24418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:30.726541 24412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:30.729461 24101 server_base.cc:1034] running on GCE node
I20260810 07:25:30.729629 24101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:30.729924 24101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:30.731076 24101 hybrid_clock.cc:648] HybridClock initialized: now 1786346730731062 us; error 32 us; skew 500 ppm
Aug 10 07:25:30 dist-test-slave-ffrd krb5kdc[23110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346730, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:30.732748 24101 init.cc:377] Logged in from keytab as oryx/127.12.157.252@KRBTEST.COM (short username oryx)
I20260810 07:25:30.733268 24101 webserver.cc:460] Webserver started at http://127.12.157.252:33101/ using document root <none> and password file <none>
I20260810 07:25:30.733479 24101 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:30.733528 24101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:30.734968 24101 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:30.736388 24427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:30.737498 24101 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 07:25:30.737581 24101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-2/wal
uuid: "1115ade1c930472ea39553bfd9614863"
format_stamp: "Formatted at 2026-08-10 07:25:30 on dist-test-slave-ffrd"
server_key: "08f8c8bfe94131710fe9ecd0927a90ab"
server_key_iv: "81fbc06280a9c1d2170e3cbaeb3ed440"
server_key_version: "encryptionkey@0"
I20260810 07:25:30.737666 24101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:30.743479 24101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:30.746004 24101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:30.751427 24101 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.252:39867
I20260810 07:25:30.751456 24515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.252:39867 every 8 connection(s)
I20260810 07:25:30.752050 24101 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/master-2/data/info.pb
I20260810 07:25:30.754585 24516 sys_catalog.cc:263] Verifying existing consensus state
I20260810 07:25:30.756181 24516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863: Bootstrap starting.
I20260810 07:25:30.768744 24516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 07:25:30.769115 24516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863: Bootstrap complete.
I20260810 07:25:30.769760 24516 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: NON_VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: true } }
I20260810 07:25:30.769917 24516 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1115ade1c930472ea39553bfd9614863, State: Initialized, Role: LEARNER
I20260810 07:25:30.770051 24516 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: NON_VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: true } }
I20260810 07:25:30.770586 24516 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:25:30.770886 24519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: NON_VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: true } } }
I20260810 07:25:30.771085 24519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [sys.catalog]: This master's current role is: LEARNER
I20260810 07:25:30.775187 24533 catalog_manager.cc:1259] Loaded cluster ID: 45c8a83eed7741dfa445f32c118347c1
I20260810 07:25:30.775250 24533 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863: loading cluster ID for follower catalog manager: success
I20260810 07:25:30.777295 24533 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863: acquiring CA information for follower catalog manager: success
I20260810 07:25:30.778623 24533 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 07:25:30.795804 24386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:30.796104 24386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:30.796200 24386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:30.800824 24386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:30.800930 24386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:30.801044 24386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:25:30.806485 24386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:43127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.193.keytab
--principal=oryx/127.12.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:36441
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=46827
--tserver_master_addrs=127.12.157.254:46027,127.12.157.253:33175,127.12.157.252:39867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:30.807612 24386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:30.808637 24386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 07:25:30.818687 24386 server_base.cc:1034] running on GCE node
W20260810 07:25:30.818624 24543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:30.818624 24544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:30.818627 24549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:30.819360 24386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:30.819970 24386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:30.821117 24386 hybrid_clock.cc:648] HybridClock initialized: now 1786346730821104 us; error 26 us; skew 500 ppm
Aug 10 07:25:30 dist-test-slave-ffrd krb5kdc[23110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346730, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:30.823860 24386 init.cc:377] Logged in from keytab as oryx/127.12.157.193@KRBTEST.COM (short username oryx)
I20260810 07:25:30.824687 24386 webserver.cc:460] Webserver started at http://127.12.157.193:46827/ using document root <none> and password file <none>
I20260810 07:25:30.825284 24386 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:30.825392 24386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:30.829067 24386 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260810 07:25:30.831018 24563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:30.832237 24386 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 07:25:30.832363 24386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "373169c708e84b2c9a7e52676b00de91"
format_stamp: "Formatted at 2026-08-10 07:25:29 on dist-test-slave-ffrd"
server_key: "947b8081c1d4dd4837a903d85eccf68e"
server_key_iv: "e0ecf3ec8af1199b39a45a18d6da456a"
server_key_version: "encryptionkey@0"
I20260810 07:25:30.832890 24386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:30.852842 24386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:30.855996 24386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:30.856158 24386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:30.856702 24386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:30.857640 24386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:30.857689 24386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:30.857731 24386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:30.857748 24386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:30.867488 24386 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:36441
I20260810 07:25:30.867555 24778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:36441 every 8 connection(s)
I20260810 07:25:30.868574 24386 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:25:30.869971 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 24386
I20260810 07:25:30.870187 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 23494
I20260810 07:25:30.904259 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:37871
--local_ip_for_outbound_sockets=127.12.157.194
--tserver_master_addrs=127.12.157.254:46027
--webserver_port=35303
--webserver_interface=127.12.157.194
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.194.keytab
--principal=oryx/127.12.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:43127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.157.254:46027,127.12.157.253:33175
--tserver_master_addrs=127.12.157.254:46027,127.12.157.253:33175,127.12.157.252:39867 with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 07:25:30 dist-test-slave-ffrd krb5kdc[23110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346730, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.253@KRBTEST.COM
Aug 10 07:25:30 dist-test-slave-ffrd krb5kdc[23110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346730, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM
Aug 10 07:25:30 dist-test-slave-ffrd krb5kdc[23110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346730, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.252@KRBTEST.COM
I20260810 07:25:30.914147 24347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.875716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:40263 (local address 127.12.157.253:33175)
0810 07:25:30.879587 (+ 3871us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:30.879594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:30.897380 (+ 17786us) server_negotiation.cc:407] Connection header received
0810 07:25:30.899071 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:30.899075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:30.899144 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:30.899207 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:30.901560 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.902392 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.904909 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.905276 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.905379 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.908750 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:30.908754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:30.908756 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:30.908823 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:30.910844 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.912428 (+ 1584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.912431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.912433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.912529 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.912843 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.912849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.912852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.913203 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:30.913290 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:30.913825 (+ 535us) server_negotiation.cc:299] Negotiation successful
0810 07:25:30.913963 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3737}
I20260810 07:25:30.914255 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.875752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:40323 (local address 127.12.157.254:46027)
0810 07:25:30.875896 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:30.875900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:30.897848 (+ 21948us) server_negotiation.cc:407] Connection header received
0810 07:25:30.899244 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:30.899247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:30.899322 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:30.899380 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:30.901560 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.902400 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.904584 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.905028 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.905158 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.909087 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:30.909094 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:30.909099 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:30.909132 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:30.910818 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.911992 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.911998 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.912001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.912106 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.912576 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.912582 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.912584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.912893 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:30.913044 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:30.913861 (+ 817us) server_negotiation.cc:299] Negotiation successful
0810 07:25:30.914018 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 07:25:30.914784 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.897941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:41857 (local address 127.12.157.252:39867)
0810 07:25:30.898821 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:30.898825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:30.898841 (+ 16us) server_negotiation.cc:407] Connection header received
0810 07:25:30.899125 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:30.899131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:30.899206 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:30.900248 (+ 1042us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:30.901550 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.902706 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.905015 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.905365 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.905725 (+ 360us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.909820 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:30.909825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:30.909829 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:30.909864 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:30.912298 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.913051 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.913054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.913056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.913138 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:30.913458 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:30.913462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:30.913463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:30.913671 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:30.913756 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:30.914087 (+ 331us) server_negotiation.cc:299] Negotiation successful
0810 07:25:30.914609 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":793,"thread_start_us":87,"threads_started":1}
I20260810 07:25:30.916086 24785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.870913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.12.157.193:40323)
0810 07:25:30.897749 (+ 26836us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:30.897765 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:30.899113 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:30.899567 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:30.899577 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:30.900321 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:30.901368 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.901390 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.902589 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.902593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.904269 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.904276 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.905369 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.905376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.906493 (+ 1117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.907202 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:30.907214 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:30.908885 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:30.911520 (+ 2635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.911523 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.911525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.911777 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.912239 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.912245 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.912250 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.912450 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.913110 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:30.913121 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:30.913524 (+ 403us) client_negotiation.cc:769] Sending connection context
0810 07:25:30.914485 (+ 961us) client_negotiation.cc:240] Negotiation successful
0810 07:25:30.914984 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26724,"mutex_wait_us":26529,"spinlock_wait_cycles":5632,"thread_start_us":244,"threads_started":1}
I20260810 07:25:30.916373 24783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.870912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:33175 (local address 127.12.157.193:40263)
0810 07:25:30.897135 (+ 26223us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:30.897228 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:30.898875 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:30.899567 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:30.899577 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:30.900335 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:30.901367 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.901390 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.902589 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.902593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.904782 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.904788 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.905526 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.905529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.905695 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.906501 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:30.906529 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:30.908499 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:30.910986 (+ 2487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.910992 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.911009 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.911391 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.912605 (+ 1214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.912611 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.912613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.912704 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.913363 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:30.913365 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:30.913522 (+ 157us) client_negotiation.cc:769] Sending connection context
0810 07:25:30.914439 (+ 917us) client_negotiation.cc:240] Negotiation successful
0810 07:25:30.914884 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25851,"mutex_wait_us":1331,"spinlock_wait_cycles":622976,"thread_start_us":25640,"threads_started":1}
I20260810 07:25:30.916085 24787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:30.898168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:39867 (local address 127.12.157.193:41857)
0810 07:25:30.898504 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:30.898517 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:30.898797 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:30.899571 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:30.899580 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:30.900346 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:30.901381 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.901397 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.902850 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.902856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.904265 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:30.904276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:30.905503 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:30.905507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:30.905661 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:30.906591 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:30.906608 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:30.909532 (+ 2924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:30.912632 (+ 3100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.912635 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.912636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.912919 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.913258 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:30.913261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:30.913263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:30.913352 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:30.913824 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:30.913827 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:30.913904 (+ 77us) client_negotiation.cc:769] Sending connection context
0810 07:25:30.915177 (+ 1273us) client_negotiation.cc:240] Negotiation successful
0810 07:25:30.915324 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":710,"spinlock_wait_cycles":3328,"thread_start_us":80,"threads_started":1}
I20260810 07:25:30.917578 24781 heartbeater.cc:347] Connected to a master server at 127.12.157.253:33175
I20260810 07:25:30.917732 24780 heartbeater.cc:347] Connected to a master server at 127.12.157.254:46027
I20260810 07:25:30.917884 24781 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:30.917888 24780 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:30.918445 24779 heartbeater.cc:347] Connected to a master server at 127.12.157.252:39867
I20260810 07:25:30.918565 24779 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:30.918605 24781 heartbeater.cc:511] Master 127.12.157.253:33175 requested a full tablet report, sending...
I20260810 07:25:30.918737 24779 heartbeater.cc:511] Master 127.12.157.252:39867 requested a full tablet report, sending...
I20260810 07:25:30.919237 24780 heartbeater.cc:511] Master 127.12.157.254:46027 requested a full tablet report, sending...
I20260810 07:25:30.920193 23186 ts_manager.cc:194] Re-registered known tserver with Master: 373169c708e84b2c9a7e52676b00de91 (127.12.157.193:36441)
I20260810 07:25:30.920385 24149 ts_manager.cc:194] Registered new tserver with Master: 373169c708e84b2c9a7e52676b00de91 (127.12.157.193:36441)
I20260810 07:25:30.920394 24449 ts_manager.cc:194] Registered new tserver with Master: 373169c708e84b2c9a7e52676b00de91 (127.12.157.193:36441)
I20260810 07:25:30.921113 23186 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.157.193@KRBTEST.COM'} at 127.12.157.193:40323
I20260810 07:25:30.947072 23924 heartbeater.cc:503] Master 127.12.157.254:46027 was elected leader, sending a full tablet report...
W20260810 07:25:31.030073 24789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:31.030380 24789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:31.030452 24789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:31.034536 24789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:31.034621 24789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:31.034709 24789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:25:31.040669 24789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:43127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.194.keytab
--principal=oryx/127.12.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:37871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=35303
--tserver_master_addrs=127.12.157.254:46027,127.12.157.253:33175,127.12.157.252:39867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:31.041914 24789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:31.042996 24789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 07:25:31.050081 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.046098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32884 (local address 127.12.157.252:39867)
0810 07:25:31.046257 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:31.046262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:31.046283 (+ 21us) server_negotiation.cc:407] Connection header received
0810 07:25:31.046369 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:31.046372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:31.046440 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:31.046550 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:31.046858 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.047643 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.048713 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.049458 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.049607 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.049729 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:31.049906 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 07:25:31.049976 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 07:25:31.050112 24079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.045959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:39867 (local address 127.0.0.1:32884)
0810 07:25:31.046152 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:31.046171 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:31.046277 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:31.046595 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:31.046598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:31.046620 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:31.046717 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:31.046723 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.047774 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.047776 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:31.048570 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:31.048580 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.049613 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.049620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:31.049718 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.049755 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 07:25:31.049863 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 07:25:31.049962 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260810 07:25:31.050860 24454 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 07:25:31.051244 24519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eebcdb40543d45c5aef6817f84cef282. Latest consensus state: current_term: 1 leader_uuid: "eebcdb40543d45c5aef6817f84cef282" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: NON_VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: true } } }
I20260810 07:25:31.051414 24519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [sys.catalog]: This master's current role is: LEARNER
W20260810 07:25:31.054376 24795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:31.054386 24794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:31.054378 24799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:31.054666 24789 server_base.cc:1034] running on GCE node
I20260810 07:25:31.055065 24789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:31.055681 24789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:31.056867 24789 hybrid_clock.cc:648] HybridClock initialized: now 1786346731056854 us; error 25 us; skew 500 ppm
Aug 10 07:25:31 dist-test-slave-ffrd krb5kdc[23110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346731, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:31.059835 24789 init.cc:377] Logged in from keytab as oryx/127.12.157.194@KRBTEST.COM (short username oryx)
I20260810 07:25:31.060768 24789 webserver.cc:460] Webserver started at http://127.12.157.194:35303/ using document root <none> and password file <none>
I20260810 07:25:31.061405 24789 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:31.061483 24789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:31.065414 24789 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 07:25:31.067667 24813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:31.068926 24789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 07:25:31.069124 24789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "313322cdeb7048778cb6dedb8a4aa76e"
format_stamp: "Formatted at 2026-08-10 07:25:29 on dist-test-slave-ffrd"
server_key: "e81d3b6b388e1c68bda448c271b84edf"
server_key_iv: "95b2430bfffee35e45f7888e74708f66"
server_key_version: "encryptionkey@0"
I20260810 07:25:31.069687 24789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:31.082258 24789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:31.085237 24789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:31.085441 24789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:31.086020 24789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:31.087010 24789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:31.087060 24789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:31.087100 24789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:31.087150 24789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:31.097728 24789 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:37871
I20260810 07:25:31.097746 25013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:37871 every 8 connection(s)
I20260810 07:25:31.098822 24789 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:25:31.105167 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 24789
I20260810 07:25:31.105367 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 23712
Aug 10 07:25:31 dist-test-slave-ffrd krb5kdc[23110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346731, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM
Aug 10 07:25:31 dist-test-slave-ffrd krb5kdc[23110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346731, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.252@KRBTEST.COM
Aug 10 07:25:31 dist-test-slave-ffrd krb5kdc[23110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346731, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.253@KRBTEST.COM
I20260810 07:25:31.118889 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.100817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:43619 (local address 127.12.157.254:46027)
0810 07:25:31.100980 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:31.100985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:31.102265 (+ 1280us) server_negotiation.cc:407] Connection header received
0810 07:25:31.103518 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:31.103523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:31.103589 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:31.103643 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:31.105118 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.105948 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.108157 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.108624 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.108727 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.113181 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:31.113189 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:31.113195 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:31.113238 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:31.115588 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:31.116519 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:31.116523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:31.116525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:31.116624 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:31.117375 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:31.117381 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:31.117386 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:31.117646 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:31.117729 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:31.118480 (+ 751us) server_negotiation.cc:299] Negotiation successful
0810 07:25:31.118659 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 07:25:31.119940 25021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.101416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.12.157.194:43619)
0810 07:25:31.102081 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:31.102145 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:31.103269 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:31.103815 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:31.103818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:31.104303 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:31.104900 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:31.104911 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.106148 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.106150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:31.108009 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:31.108017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.108932 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.108936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:31.109307 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.110835 (+ 1528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:31.110850 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:31.112870 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:31.115772 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:31.115779 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:31.115806 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:31.116395 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:31.116971 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:31.116978 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:31.116983 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:31.117236 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:31.117798 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:31.117807 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:31.118163 (+ 356us) client_negotiation.cc:769] Sending connection context
0810 07:25:31.119174 (+ 1011us) client_negotiation.cc:240] Negotiation successful
0810 07:25:31.119399 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":499,"spinlock_wait_cycles":83584,"thread_start_us":160,"threads_started":1}
I20260810 07:25:31.120287 25019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.101416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:39867 (local address 127.12.157.194:59699)
0810 07:25:31.102081 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:31.102145 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:31.103269 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:31.103876 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:31.103879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:31.104309 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:31.104908 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:31.104915 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.106135 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.106139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:31.108005 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:31.108017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.108932 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.108965 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:31.109355 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.110483 (+ 1128us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:31.110522 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:31.113375 (+ 2853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:31.116137 (+ 2762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:31.116141 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:31.116143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:31.116791 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:31.117149 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:31.117155 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:31.117160 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:31.117258 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:31.117885 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:31.117891 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:31.118170 (+ 279us) client_negotiation.cc:769] Sending connection context
0810 07:25:31.118432 (+ 262us) client_negotiation.cc:240] Negotiation successful
0810 07:25:31.118851 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":443,"spinlock_wait_cycles":37120,"thread_start_us":134,"threads_started":1}
I20260810 07:25:31.121009 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.100817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:59699 (local address 127.12.157.252:39867)
0810 07:25:31.100983 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:31.100987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:31.102264 (+ 1277us) server_negotiation.cc:407] Connection header received
0810 07:25:31.103605 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:31.103610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:31.103671 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:31.103744 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:31.105108 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.105948 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.108156 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.108624 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.108739 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.113571 (+ 4832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:31.113576 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:31.113890 (+ 314us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:31.113935 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:31.115999 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:31.116907 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:31.116911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:31.116913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:31.117035 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:31.117367 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:31.117374 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:31.117378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:31.117684 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:31.117822 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:31.120621 (+ 2799us) server_negotiation.cc:299] Negotiation successful
0810 07:25:31.120781 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 07:25:31.122156 25014 heartbeater.cc:347] Connected to a master server at 127.12.157.252:39867
I20260810 07:25:31.122398 25014 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:31.122969 25014 heartbeater.cc:511] Master 127.12.157.252:39867 requested a full tablet report, sending...
I20260810 07:25:31.124212 24347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.102673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:57383 (local address 127.12.157.253:33175)
0810 07:25:31.102806 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:31.102816 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:31.102900 (+ 84us) server_negotiation.cc:407] Connection header received
0810 07:25:31.103493 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:31.103495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:31.103553 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:31.103629 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:31.105098 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.105963 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.108181 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.108682 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.108782 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.113680 (+ 4898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:31.113685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:31.113688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:31.113721 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:31.115999 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:31.116650 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:31.116657 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:31.116662 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:31.116790 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:31.121766 (+ 4976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:31.121773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:31.121777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:31.122035 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:31.122106 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:31.123855 (+ 1749us) server_negotiation.cc:299] Negotiation successful
0810 07:25:31.123995 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 07:25:31.124401 24449 ts_manager.cc:194] Registered new tserver with Master: 313322cdeb7048778cb6dedb8a4aa76e (127.12.157.194:37871)
I20260810 07:25:31.124833 25022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.102569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:33175 (local address 127.12.157.194:57383)
0810 07:25:31.102810 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:31.102838 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:31.103298 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:31.103810 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:31.103817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:31.104288 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:31.104875 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:31.104898 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.106146 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.106153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:31.108014 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:31.108029 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.109064 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.109071 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:31.109533 (+ 462us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.110488 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:31.110521 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:31.113520 (+ 2999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:31.116137 (+ 2617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:31.116141 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:31.116143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:31.116528 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:31.121408 (+ 4880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:31.121414 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:31.121418 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:31.121646 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:31.123544 (+ 1898us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:31.123549 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:31.123655 (+ 106us) client_negotiation.cc:769] Sending connection context
0810 07:25:31.124400 (+ 745us) client_negotiation.cc:240] Negotiation successful
0810 07:25:31.124570 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":322,"spinlock_wait_cycles":19840,"thread_start_us":84,"threads_started":1}
I20260810 07:25:31.130872 25015 heartbeater.cc:347] Connected to a master server at 127.12.157.254:46027
I20260810 07:25:31.130878 25017 heartbeater.cc:347] Connected to a master server at 127.12.157.253:33175
I20260810 07:25:31.130999 25015 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:31.131035 25017 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:31.131261 25017 heartbeater.cc:511] Master 127.12.157.253:33175 requested a full tablet report, sending...
I20260810 07:25:31.131269 25015 heartbeater.cc:511] Master 127.12.157.254:46027 requested a full tablet report, sending...
I20260810 07:25:31.132094 23186 ts_manager.cc:194] Re-registered known tserver with Master: 313322cdeb7048778cb6dedb8a4aa76e (127.12.157.194:37871)
I20260810 07:25:31.132138 24149 ts_manager.cc:194] Registered new tserver with Master: 313322cdeb7048778cb6dedb8a4aa76e (127.12.157.194:37871)
I20260810 07:25:31.133087 23186 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.157.194@KRBTEST.COM'} at 127.12.157.194:43619
I20260810 07:25:31.137100 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:32917
--local_ip_for_outbound_sockets=127.12.157.195
--tserver_master_addrs=127.12.157.254:46027
--webserver_port=33401
--webserver_interface=127.12.157.195
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.195.keytab
--principal=oryx/127.12.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:43127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.157.254:46027,127.12.157.253:33175
--tserver_master_addrs=127.12.157.254:46027,127.12.157.253:33175,127.12.157.252:39867 with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:31.256790 25024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:31.257113 25024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:31.257164 25024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:31.261039 25024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:31.261086 25024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:31.261140 25024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:25:31.266352 25024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:43127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.195.keytab
--principal=oryx/127.12.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:32917
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=33401
--tserver_master_addrs=127.12.157.254:46027,127.12.157.253:33175,127.12.157.252:39867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:31.267436 25024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:31.268508 25024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:31.278949 25037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:31.278949 25032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:31.278976 25041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:31.279110 25024 server_base.cc:1034] running on GCE node
I20260810 07:25:31.279556 25024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:31.280182 25024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:31.281376 25024 hybrid_clock.cc:648] HybridClock initialized: now 1786346731281358 us; error 41 us; skew 500 ppm
Aug 10 07:25:31 dist-test-slave-ffrd krb5kdc[23110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346731, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:31.284238 25024 init.cc:377] Logged in from keytab as oryx/127.12.157.195@KRBTEST.COM (short username oryx)
I20260810 07:25:31.285113 25024 webserver.cc:460] Webserver started at http://127.12.157.195:33401/ using document root <none> and password file <none>
I20260810 07:25:31.285632 25024 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:31.285706 25024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:31.289446 25024 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260810 07:25:31.291483 25050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:31.292773 25024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 07:25:31.292959 25024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "862aee2e55284192955041c46f55f8ea"
format_stamp: "Formatted at 2026-08-10 07:25:29 on dist-test-slave-ffrd"
server_key: "4415c2117a813753cd6c358458375783"
server_key_iv: "6c947acc330ddea6db4e2ecce54a1eec"
server_key_version: "encryptionkey@0"
I20260810 07:25:31.293562 25024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:31.313686 25024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:31.316888 25024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:31.317175 25024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:31.317848 25024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:31.318746 25024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:31.318822 25024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:31.318897 25024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:31.318928 25024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:31.328771 25024 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:32917
I20260810 07:25:31.328799 25232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:32917 every 8 connection(s)
I20260810 07:25:31.329979 25024 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:25:31.334919 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 25024
Aug 10 07:25:31 dist-test-slave-ffrd krb5kdc[23110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346731, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM
Aug 10 07:25:31 dist-test-slave-ffrd krb5kdc[23110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346731, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.253@KRBTEST.COM
Aug 10 07:25:31 dist-test-slave-ffrd krb5kdc[23110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346731, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.252@KRBTEST.COM
I20260810 07:25:31.350441 25255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.338126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56470 (local address 127.12.157.253:33175)
0810 07:25:31.339221 (+ 1095us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:31.339225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:31.339656 (+ 431us) server_negotiation.cc:407] Connection header received
0810 07:25:31.339695 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:31.339697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:31.339749 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:31.339825 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:31.340865 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.341684 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.343008 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.343362 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.343461 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.345865 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:31.345869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:31.345873 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:31.345905 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:31.347848 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:31.348325 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:31.348329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:31.348331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:31.348421 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:31.348713 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:31.348716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:31.348718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:31.348978 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:31.349131 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:31.350049 (+ 918us) server_negotiation.cc:299] Negotiation successful
0810 07:25:31.350192 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":520,"server-negotiator.queue_time_us":1022,"thread_start_us":78,"threads_started":1}
I20260810 07:25:31.353664 25254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.337641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52606 (local address 127.12.157.254:46027)
0810 07:25:31.339300 (+ 1659us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:31.339307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:31.339324 (+ 17us) server_negotiation.cc:407] Connection header received
0810 07:25:31.339371 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:31.339376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:31.339430 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:31.340628 (+ 1198us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:31.340774 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.341748 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.343097 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.343434 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.343514 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.345994 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:31.345998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:31.346000 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:31.346027 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:31.351189 (+ 5162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:31.351779 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:31.351783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:31.351786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:31.351885 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:31.352250 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:31.352253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:31.352255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:31.352507 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:31.353243 (+ 736us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:31.353326 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 07:25:31.353467 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1399,"thread_start_us":81,"threads_started":1}
I20260810 07:25:31.354477 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.331938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:44129 (local address 127.12.157.254:46027)
0810 07:25:31.332110 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:31.332117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:31.333063 (+ 946us) server_negotiation.cc:407] Connection header received
0810 07:25:31.334349 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:31.334353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:31.334420 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:31.334536 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:31.336192 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.337128 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.339398 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.339781 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.339906 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.345023 (+ 5117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:31.345028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:31.345031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:31.345061 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:31.348856 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:31.351964 (+ 3108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:31.351968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:31.351971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:31.352069 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:31.352580 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:31.352585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:31.352588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:31.352868 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:31.353012 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:31.354184 (+ 1172us) server_negotiation.cc:299] Negotiation successful
0810 07:25:31.354309 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 07:25:31.355060 25242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.332319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.12.157.195:44129)
0810 07:25:31.332858 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:31.332917 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:31.334117 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:31.334735 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:31.334742 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:31.335324 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:31.335984 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:31.336009 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.337329 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.337332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:31.339249 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:31.339271 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.340095 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.340103 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:31.340420 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.341428 (+ 1008us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:31.341444 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:31.344819 (+ 3375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:31.351394 (+ 6575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:31.351406 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:31.351428 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:31.351833 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:31.352207 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:31.352211 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:31.352214 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:31.352450 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:31.353072 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:31.353079 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:31.353408 (+ 329us) client_negotiation.cc:769] Sending connection context
0810 07:25:31.353604 (+ 196us) client_negotiation.cc:240] Negotiation successful
0810 07:25:31.353977 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":644,"spinlock_wait_cycles":13056,"thread_start_us":120,"threads_started":1}
I20260810 07:25:31.355068 25244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.332319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:39867 (local address 127.12.157.195:39029)
0810 07:25:31.333118 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:31.333134 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:31.334131 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:31.336192 (+ 2061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:31.336195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:31.336420 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:31.336546 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:31.336554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.337731 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.337735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:31.342825 (+ 5090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:31.342837 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.343866 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.343870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:31.343983 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.344807 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:31.344822 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:31.347267 (+ 2445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:31.351969 (+ 4702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:31.351975 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:31.351981 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:31.352244 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:31.352607 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:31.352614 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:31.352619 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:31.352714 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:31.353293 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:31.353296 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:31.353423 (+ 127us) client_negotiation.cc:769] Sending connection context
0810 07:25:31.353604 (+ 181us) client_negotiation.cc:240] Negotiation successful
0810 07:25:31.354044 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"mutex_wait_us":1434,"spinlock_wait_cycles":40576,"thread_start_us":164,"threads_started":1}
I20260810 07:25:31.356683 25247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.333296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:33175 (local address 127.12.157.195:37607)
0810 07:25:31.333600 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:31.333614 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:31.334131 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:31.334739 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:31.334745 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:31.335345 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:31.335983 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:31.336010 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.337337 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.337340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:31.339203 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:31.339218 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.340095 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.340101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:31.340419 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.341276 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:31.341298 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:31.348011 (+ 6713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:31.353838 (+ 5827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:31.353841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:31.353843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:31.354460 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:31.355404 (+ 944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:31.355407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:31.355409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:31.355504 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:31.356026 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:31.356030 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:31.356135 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 07:25:31.356332 (+ 197us) client_negotiation.cc:240] Negotiation successful
0810 07:25:31.356481 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":199,"spinlock_wait_cycles":7808,"thread_start_us":95,"threads_started":1}
I20260810 07:25:31.356739 24347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.333686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:37607 (local address 127.12.157.253:33175)
0810 07:25:31.333847 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:31.333852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:31.333870 (+ 18us) server_negotiation.cc:407] Connection header received
0810 07:25:31.334361 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:31.334365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:31.334420 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:31.334477 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:31.336268 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.337071 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.339357 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.339699 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.339815 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.348194 (+ 8379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:31.348199 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:31.348202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:31.348232 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:31.350159 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:31.355145 (+ 4986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:31.355149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:31.355151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:31.355251 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:31.355618 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:31.355621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:31.355624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:31.355850 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:31.355940 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:31.356413 (+ 473us) server_negotiation.cc:299] Negotiation successful
0810 07:25:31.356558 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":71}
I20260810 07:25:31.357298 25238 heartbeater.cc:347] Connected to a master server at 127.12.157.254:46027
I20260810 07:25:31.357539 25240 heartbeater.cc:347] Connected to a master server at 127.12.157.253:33175
I20260810 07:25:31.357564 25238 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:31.357689 25240 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:31.358290 25238 heartbeater.cc:511] Master 127.12.157.254:46027 requested a full tablet report, sending...
I20260810 07:25:31.358316 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.331865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:39029 (local address 127.12.157.252:39867)
0810 07:25:31.331997 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:31.332001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:31.333240 (+ 1239us) server_negotiation.cc:407] Connection header received
0810 07:25:31.334374 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:31.334377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:31.334434 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:31.334487 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:31.336701 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.337571 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.343068 (+ 5497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.343499 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.343595 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.347457 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:31.347463 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:31.347467 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:31.347495 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:31.351237 (+ 3742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:31.352383 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:31.352387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:31.352388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:31.352476 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:31.352868 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:31.352872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:31.352874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:31.353140 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:31.357904 (+ 4764us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:31.358000 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 07:25:31.358145 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 07:25:31.358297 25240 heartbeater.cc:511] Master 127.12.157.253:33175 requested a full tablet report, sending...
I20260810 07:25:31.358903 25236 heartbeater.cc:347] Connected to a master server at 127.12.157.252:39867
I20260810 07:25:31.358975 25236 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:31.359110 25236 heartbeater.cc:511] Master 127.12.157.252:39867 requested a full tablet report, sending...
I20260810 07:25:31.359560 24149 ts_manager.cc:194] Registered new tserver with Master: 862aee2e55284192955041c46f55f8ea (127.12.157.195:32917)
I20260810 07:25:31.359764 24449 ts_manager.cc:194] Registered new tserver with Master: 862aee2e55284192955041c46f55f8ea (127.12.157.195:32917)
I20260810 07:25:31.362054 25258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.338813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32894 (local address 127.12.157.252:39867)
0810 07:25:31.344242 (+ 5429us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:31.344247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:31.344264 (+ 17us) server_negotiation.cc:407] Connection header received
0810 07:25:31.344306 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:31.344311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:31.344364 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:31.344460 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:31.346724 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.347554 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.348715 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.349084 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.349175 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.351092 (+ 1917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:31.351096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:31.351099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:31.351126 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:31.360279 (+ 9153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:31.360790 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:31.360793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:31.360795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:31.360880 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:31.361223 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:31.361226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:31.361227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:31.361403 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:31.361505 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:31.361777 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 07:25:31.361892 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":95,"threads_started":1}
I20260810 07:25:31.358548 23186 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:25:31.363168 23186 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:25:31.365289 23185 ts_manager.cc:194] Re-registered known tserver with Master: 862aee2e55284192955041c46f55f8ea (127.12.157.195:32917)
I20260810 07:25:31.365955 24391 sys_catalog.cc:455] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eebcdb40543d45c5aef6817f84cef282" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: NON_VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: true } } }
I20260810 07:25:31.366045 23185 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.157.195@KRBTEST.COM'} at 127.12.157.195:44129
I20260810 07:25:31.366082 24391 sys_catalog.cc:458] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:31.370172 25264 raft_consensus.cc:1050] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282: attempting to promote NON_VOTER 1115ade1c930472ea39553bfd9614863 to VOTER
I20260810 07:25:31.370474 25264 consensus_queue.cc:237] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: false } }
I20260810 07:25:31.371698 24162 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [term 1 FOLLOWER]: Refusing update from remote peer eebcdb40543d45c5aef6817f84cef282: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 07:25:31.372092 24391 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 07:25:31.372239 24473 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [term 1 LEARNER]: Refusing update from remote peer eebcdb40543d45c5aef6817f84cef282: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 07:25:31.372761 24391 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 07:25:31.376473 25264 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 1115ade1c930472ea39553bfd9614863 (127.12.157.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: false } } }
I20260810 07:25:31.377043 24162 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 1115ade1c930472ea39553bfd9614863 (127.12.157.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: false } } }
I20260810 07:25:31.378170 25264 sys_catalog.cc:455] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "eebcdb40543d45c5aef6817f84cef282" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: false } } }
I20260810 07:25:31.378300 25264 sys_catalog.cc:458] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:31.378034 24473 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 1115ade1c930472ea39553bfd9614863 (127.12.157.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: false } } }
I20260810 07:25:31.378460 25264 sys_catalog.cc:455] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eebcdb40543d45c5aef6817f84cef282" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: false } } }
I20260810 07:25:31.378559 25264 sys_catalog.cc:458] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:31.378648 25264 sys_catalog.cc:455] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eebcdb40543d45c5aef6817f84cef282" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: false } } }
I20260810 07:25:31.378716 25264 sys_catalog.cc:458] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:31.379752 25279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eebcdb40543d45c5aef6817f84cef282" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: false } } }
I20260810 07:25:31.379902 25279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:25:31.380998 24519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eebcdb40543d45c5aef6817f84cef282" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: false } } }
I20260810 07:25:31.381131 24519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:25:31.388190 24079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.382845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:37871 (local address 127.0.0.1:60380)
0810 07:25:31.382989 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:31.383007 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:31.383100 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:31.384083 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:31.384087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:31.384108 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:31.384237 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:31.384245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.385813 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.385816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:31.386574 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:31.386583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.387743 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.387746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:31.387835 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.387865 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 07:25:31.387911 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 07:25:31.388008 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 07:25:31.388919 25280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.383064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60380 (local address 127.12.157.194:37871)
0810 07:25:31.383480 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:31.383495 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:31.383525 (+ 30us) server_negotiation.cc:407] Connection header received
0810 07:25:31.383621 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:31.383627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:31.383886 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:31.384413 (+ 527us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:31.384572 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.385720 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.386712 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.387599 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.388440 (+ 841us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.388604 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:31.388659 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 07:25:31.388754 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":164,"threads_started":1}
I20260810 07:25:31.389581 25281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.383466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:32917 (local address 127.0.0.1:56216)
0810 07:25:31.384440 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:31.384451 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:31.384546 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:31.385335 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:31.385338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:31.385357 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:31.385455 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:31.385465 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.386811 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.386815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:31.387924 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:31.387931 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.388974 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.388979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:31.389281 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.389306 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 07:25:31.389378 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 07:25:31.389451 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":919,"mutex_wait_us":1,"thread_start_us":114,"threads_started":1}
I20260810 07:25:31.389757 25285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.384074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60778 (local address 127.12.157.193:36441)
0810 07:25:31.384880 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:31.384897 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:31.384920 (+ 23us) server_negotiation.cc:407] Connection header received
0810 07:25:31.385027 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:31.385033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:31.385213 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:31.385365 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:31.385752 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.386839 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.388186 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.389188 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.389339 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.389479 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:31.389536 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 07:25:31.389625 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":84,"threads_started":1}
I20260810 07:25:31.389748 25284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.383152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:36441 (local address 127.0.0.1:60778)
0810 07:25:31.384041 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:31.384054 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:31.384157 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:31.385517 (+ 1360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:31.385520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:31.385536 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:31.385621 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:31.385627 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.386931 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.386937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:31.388046 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:31.388056 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.389324 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.389328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:31.389414 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.389437 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 07:25:31.389506 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 07:25:31.389581 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"mutex_wait_us":1,"thread_start_us":1149,"threads_started":1}
I20260810 07:25:31.392481 24659 tablet_service.cc:1467] Processing CreateTablet for tablet 4084eb94554d43bfb0230a39e8621cc7 (DEFAULT_TABLE table=test-table [id=0d928fbabeda4b5da0555da738f312de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:31.392481 24901 tablet_service.cc:1467] Processing CreateTablet for tablet 4084eb94554d43bfb0230a39e8621cc7 (DEFAULT_TABLE table=test-table [id=0d928fbabeda4b5da0555da738f312de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:31.393673 25286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.384056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56216 (local address 127.12.157.195:32917)
0810 07:25:31.384826 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:31.384839 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:31.384865 (+ 26us) server_negotiation.cc:407] Connection header received
0810 07:25:31.384975 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:31.384980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:31.385181 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:31.385351 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:31.385564 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.386711 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.388029 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.388812 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.388987 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.393328 (+ 4341us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:31.393389 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 07:25:31.393505 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":137,"threads_started":1}
I20260810 07:25:31.393738 24901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4084eb94554d43bfb0230a39e8621cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:31.393738 24659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4084eb94554d43bfb0230a39e8621cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:31.396579 25130 tablet_service.cc:1467] Processing CreateTablet for tablet 4084eb94554d43bfb0230a39e8621cc7 (DEFAULT_TABLE table=test-table [id=0d928fbabeda4b5da0555da738f312de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:31.398224 25130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4084eb94554d43bfb0230a39e8621cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:31.402828 25290 tablet_bootstrap.cc:492] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91: Bootstrap starting.
I20260810 07:25:31.403541 25289 tablet_bootstrap.cc:492] T 4084eb94554d43bfb0230a39e8621cc7 P 313322cdeb7048778cb6dedb8a4aa76e: Bootstrap starting.
I20260810 07:25:31.406155 25289 tablet_bootstrap.cc:654] T 4084eb94554d43bfb0230a39e8621cc7 P 313322cdeb7048778cb6dedb8a4aa76e: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:31.406277 25292 tablet_bootstrap.cc:492] T 4084eb94554d43bfb0230a39e8621cc7 P 862aee2e55284192955041c46f55f8ea: Bootstrap starting.
I20260810 07:25:31.407346 25290 tablet_bootstrap.cc:654] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:31.407337 25289 log.cc:824] T 4084eb94554d43bfb0230a39e8621cc7 P 313322cdeb7048778cb6dedb8a4aa76e: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:31.408337 25290 log.cc:824] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:31.408552 25292 tablet_bootstrap.cc:654] T 4084eb94554d43bfb0230a39e8621cc7 P 862aee2e55284192955041c46f55f8ea: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:31.409413 25292 log.cc:824] T 4084eb94554d43bfb0230a39e8621cc7 P 862aee2e55284192955041c46f55f8ea: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:31.409930 25289 tablet_bootstrap.cc:492] T 4084eb94554d43bfb0230a39e8621cc7 P 313322cdeb7048778cb6dedb8a4aa76e: No bootstrap required, opened a new log
I20260810 07:25:31.410171 25289 ts_tablet_manager.cc:1397] T 4084eb94554d43bfb0230a39e8621cc7 P 313322cdeb7048778cb6dedb8a4aa76e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 07:25:31.411547 25290 tablet_bootstrap.cc:492] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91: No bootstrap required, opened a new log
I20260810 07:25:31.411768 25290 ts_tablet_manager.cc:1397] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260810 07:25:31.412010 25292 tablet_bootstrap.cc:492] T 4084eb94554d43bfb0230a39e8621cc7 P 862aee2e55284192955041c46f55f8ea: No bootstrap required, opened a new log
I20260810 07:25:31.412202 25292 ts_tablet_manager.cc:1397] T 4084eb94554d43bfb0230a39e8621cc7 P 862aee2e55284192955041c46f55f8ea: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 07:25:31.414506 25289 raft_consensus.cc:348] T 4084eb94554d43bfb0230a39e8621cc7 P 313322cdeb7048778cb6dedb8a4aa76e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313322cdeb7048778cb6dedb8a4aa76e" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37871 } } peers { permanent_uuid: "373169c708e84b2c9a7e52676b00de91" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 36441 } } peers { permanent_uuid: "862aee2e55284192955041c46f55f8ea" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 32917 } }
I20260810 07:25:31.414798 25289 raft_consensus.cc:374] T 4084eb94554d43bfb0230a39e8621cc7 P 313322cdeb7048778cb6dedb8a4aa76e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:31.414865 25289 raft_consensus.cc:729] T 4084eb94554d43bfb0230a39e8621cc7 P 313322cdeb7048778cb6dedb8a4aa76e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 313322cdeb7048778cb6dedb8a4aa76e, State: Initialized, Role: FOLLOWER
I20260810 07:25:31.415410 25289 consensus_queue.cc:260] T 4084eb94554d43bfb0230a39e8621cc7 P 313322cdeb7048778cb6dedb8a4aa76e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313322cdeb7048778cb6dedb8a4aa76e" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37871 } } peers { permanent_uuid: "373169c708e84b2c9a7e52676b00de91" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 36441 } } peers { permanent_uuid: "862aee2e55284192955041c46f55f8ea" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 32917 } }
I20260810 07:25:31.416254 25015 heartbeater.cc:503] Master 127.12.157.254:46027 was elected leader, sending a full tablet report...
I20260810 07:25:31.418149 25292 raft_consensus.cc:348] T 4084eb94554d43bfb0230a39e8621cc7 P 862aee2e55284192955041c46f55f8ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313322cdeb7048778cb6dedb8a4aa76e" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37871 } } peers { permanent_uuid: "373169c708e84b2c9a7e52676b00de91" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 36441 } } peers { permanent_uuid: "862aee2e55284192955041c46f55f8ea" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 32917 } }
I20260810 07:25:31.418502 25292 raft_consensus.cc:374] T 4084eb94554d43bfb0230a39e8621cc7 P 862aee2e55284192955041c46f55f8ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:31.418562 25292 raft_consensus.cc:729] T 4084eb94554d43bfb0230a39e8621cc7 P 862aee2e55284192955041c46f55f8ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862aee2e55284192955041c46f55f8ea, State: Initialized, Role: FOLLOWER
I20260810 07:25:31.419050 25292 consensus_queue.cc:260] T 4084eb94554d43bfb0230a39e8621cc7 P 862aee2e55284192955041c46f55f8ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313322cdeb7048778cb6dedb8a4aa76e" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37871 } } peers { permanent_uuid: "373169c708e84b2c9a7e52676b00de91" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 36441 } } peers { permanent_uuid: "862aee2e55284192955041c46f55f8ea" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 32917 } }
I20260810 07:25:31.419242 25290 raft_consensus.cc:348] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313322cdeb7048778cb6dedb8a4aa76e" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37871 } } peers { permanent_uuid: "373169c708e84b2c9a7e52676b00de91" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 36441 } } peers { permanent_uuid: "862aee2e55284192955041c46f55f8ea" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 32917 } }
I20260810 07:25:31.419529 25290 raft_consensus.cc:374] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:31.419603 25290 raft_consensus.cc:729] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 373169c708e84b2c9a7e52676b00de91, State: Initialized, Role: FOLLOWER
I20260810 07:25:31.420109 25290 consensus_queue.cc:260] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313322cdeb7048778cb6dedb8a4aa76e" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37871 } } peers { permanent_uuid: "373169c708e84b2c9a7e52676b00de91" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 36441 } } peers { permanent_uuid: "862aee2e55284192955041c46f55f8ea" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 32917 } }
I20260810 07:25:31.420276 25292 ts_tablet_manager.cc:1428] T 4084eb94554d43bfb0230a39e8621cc7 P 862aee2e55284192955041c46f55f8ea: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260810 07:25:31.420656 25238 heartbeater.cc:503] Master 127.12.157.254:46027 was elected leader, sending a full tablet report...
I20260810 07:25:31.421134 25290 ts_tablet_manager.cc:1428] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260810 07:25:31.421494 24780 heartbeater.cc:503] Master 127.12.157.254:46027 was elected leader, sending a full tablet report...
I20260810 07:25:31.427081 25289 ts_tablet_manager.cc:1428] T 4084eb94554d43bfb0230a39e8621cc7 P 313322cdeb7048778cb6dedb8a4aa76e: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260810 07:25:31.554705 25298 raft_consensus.cc:482] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:25:31.554987 25298 raft_consensus.cc:504] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313322cdeb7048778cb6dedb8a4aa76e" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37871 } } peers { permanent_uuid: "373169c708e84b2c9a7e52676b00de91" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 36441 } } peers { permanent_uuid: "862aee2e55284192955041c46f55f8ea" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 32917 } }
I20260810 07:25:31.556218 25298 leader_election.cc:288] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 313322cdeb7048778cb6dedb8a4aa76e (127.12.157.194:37871), 862aee2e55284192955041c46f55f8ea (127.12.157.195:32917)
I20260810 07:25:31.561800 25280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.556708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:37123 (local address 127.12.157.194:37871)
0810 07:25:31.556866 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:31.556871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:31.557146 (+ 275us) server_negotiation.cc:407] Connection header received
0810 07:25:31.557282 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:31.557285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:31.557331 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:31.557430 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:31.557816 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.558621 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.560421 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.561063 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.561195 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.561250 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:31.561572 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 07:25:31.561658 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 07:25:31.561811 25286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.556769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:42327 (local address 127.12.157.195:32917)
0810 07:25:31.556901 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:31.556906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:31.557139 (+ 233us) server_negotiation.cc:407] Connection header received
0810 07:25:31.557274 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:31.557277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:31.557331 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:31.557430 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:31.557823 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.558621 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.560421 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.561079 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.561195 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.561250 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:31.561572 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 07:25:31.561658 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 07:25:31.561815 25301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.556550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:32917 (local address 127.12.157.193:42327)
0810 07:25:31.557023 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:31.557040 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:31.557178 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:31.557457 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:31.557460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:31.557488 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:31.557680 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:31.557687 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.558751 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.558755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:31.560249 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:31.560267 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.561201 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.561204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:31.561360 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.561430 (+ 70us) client_negotiation.cc:769] Sending connection context
0810 07:25:31.561513 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 07:25:31.561613 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":276,"spinlock_wait_cycles":39040,"thread_start_us":123,"threads_started":1}
I20260810 07:25:31.561818 25303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.556550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:37871 (local address 127.12.157.193:37123)
0810 07:25:31.557057 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:31.557072 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:31.557172 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:31.557457 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:31.557460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:31.557486 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:31.557687 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:31.557693 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.558751 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.558776 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:31.560256 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:31.560267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.561201 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.561204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:31.561339 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.561430 (+ 91us) client_negotiation.cc:769] Sending connection context
0810 07:25:31.561513 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 07:25:31.561617 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":346,"spinlock_wait_cycles":10240,"thread_start_us":49,"threads_started":1}
I20260810 07:25:31.562757 24932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4084eb94554d43bfb0230a39e8621cc7" candidate_uuid: "373169c708e84b2c9a7e52676b00de91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "313322cdeb7048778cb6dedb8a4aa76e" is_pre_election: true
I20260810 07:25:31.562755 25166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4084eb94554d43bfb0230a39e8621cc7" candidate_uuid: "373169c708e84b2c9a7e52676b00de91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862aee2e55284192955041c46f55f8ea" is_pre_election: true
I20260810 07:25:31.563092 24932 raft_consensus.cc:2454] T 4084eb94554d43bfb0230a39e8621cc7 P 313322cdeb7048778cb6dedb8a4aa76e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 373169c708e84b2c9a7e52676b00de91 in term 0.
I20260810 07:25:31.563092 25166 raft_consensus.cc:2454] T 4084eb94554d43bfb0230a39e8621cc7 P 862aee2e55284192955041c46f55f8ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 373169c708e84b2c9a7e52676b00de91 in term 0.
I20260810 07:25:31.563618 24570 leader_election.cc:302] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 313322cdeb7048778cb6dedb8a4aa76e, 373169c708e84b2c9a7e52676b00de91, 862aee2e55284192955041c46f55f8ea; no voters:
I20260810 07:25:31.563932 25298 raft_consensus.cc:2781] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:25:31.564046 25298 raft_consensus.cc:482] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:25:31.564131 25298 raft_consensus.cc:3037] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:31.565377 25298 raft_consensus.cc:504] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313322cdeb7048778cb6dedb8a4aa76e" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37871 } } peers { permanent_uuid: "373169c708e84b2c9a7e52676b00de91" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 36441 } } peers { permanent_uuid: "862aee2e55284192955041c46f55f8ea" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 32917 } }
I20260810 07:25:31.565745 25298 leader_election.cc:288] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91 [CANDIDATE]: Term 1 election: Requested vote from peers 313322cdeb7048778cb6dedb8a4aa76e (127.12.157.194:37871), 862aee2e55284192955041c46f55f8ea (127.12.157.195:32917)
I20260810 07:25:31.566133 24932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4084eb94554d43bfb0230a39e8621cc7" candidate_uuid: "373169c708e84b2c9a7e52676b00de91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "313322cdeb7048778cb6dedb8a4aa76e"
I20260810 07:25:31.566133 25166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4084eb94554d43bfb0230a39e8621cc7" candidate_uuid: "373169c708e84b2c9a7e52676b00de91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862aee2e55284192955041c46f55f8ea"
I20260810 07:25:31.566263 25166 raft_consensus.cc:3037] T 4084eb94554d43bfb0230a39e8621cc7 P 862aee2e55284192955041c46f55f8ea [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:31.566295 24932 raft_consensus.cc:3037] T 4084eb94554d43bfb0230a39e8621cc7 P 313322cdeb7048778cb6dedb8a4aa76e [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:31.567667 24932 raft_consensus.cc:2454] T 4084eb94554d43bfb0230a39e8621cc7 P 313322cdeb7048778cb6dedb8a4aa76e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 373169c708e84b2c9a7e52676b00de91 in term 1.
I20260810 07:25:31.567667 25166 raft_consensus.cc:2454] T 4084eb94554d43bfb0230a39e8621cc7 P 862aee2e55284192955041c46f55f8ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 373169c708e84b2c9a7e52676b00de91 in term 1.
I20260810 07:25:31.567970 24570 leader_election.cc:302] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 313322cdeb7048778cb6dedb8a4aa76e, 373169c708e84b2c9a7e52676b00de91, 862aee2e55284192955041c46f55f8ea; no voters:
I20260810 07:25:31.568132 25298 raft_consensus.cc:2781] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:31.568380 25298 raft_consensus.cc:686] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91 [term 1 LEADER]: Becoming Leader. State: Replica: 373169c708e84b2c9a7e52676b00de91, State: Running, Role: LEADER
I20260810 07:25:31.568771 25298 consensus_queue.cc:237] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313322cdeb7048778cb6dedb8a4aa76e" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37871 } } peers { permanent_uuid: "373169c708e84b2c9a7e52676b00de91" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 36441 } } peers { permanent_uuid: "862aee2e55284192955041c46f55f8ea" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 32917 } }
I20260810 07:25:31.573004 23186 catalog_manager.cc:5495] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 373169c708e84b2c9a7e52676b00de91 (127.12.157.193). New cstate: current_term: 1 leader_uuid: "373169c708e84b2c9a7e52676b00de91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "313322cdeb7048778cb6dedb8a4aa76e" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "373169c708e84b2c9a7e52676b00de91" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 36441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "862aee2e55284192955041c46f55f8ea" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 32917 } health_report { overall_health: UNKNOWN } } }
W20260810 07:25:31.581954 25241 tablet.cc:2369] T 4084eb94554d43bfb0230a39e8621cc7 P 862aee2e55284192955041c46f55f8ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:25:31.585883 25285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.582203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60792 (local address 127.12.157.193:36441)
0810 07:25:31.582562 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:31.582565 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:31.582578 (+ 13us) server_negotiation.cc:407] Connection header received
0810 07:25:31.582637 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:31.582640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:31.582689 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:31.582786 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:25:31.583105 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.583600 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.584428 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.584632 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.584706 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.585096 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:25:31.585494 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:25:31.585588 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:31.585661 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 07:25:31.585712 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285}
I20260810 07:25:31.591652 25166 raft_consensus.cc:1261] T 4084eb94554d43bfb0230a39e8621cc7 P 862aee2e55284192955041c46f55f8ea [term 1 FOLLOWER]: Refusing update from remote peer 373169c708e84b2c9a7e52676b00de91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:31.591673 24932 raft_consensus.cc:1261] T 4084eb94554d43bfb0230a39e8621cc7 P 313322cdeb7048778cb6dedb8a4aa76e [term 1 FOLLOWER]: Refusing update from remote peer 373169c708e84b2c9a7e52676b00de91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:31.592334 25304 consensus_queue.cc:1035] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "313322cdeb7048778cb6dedb8a4aa76e" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:31.592528 25298 consensus_queue.cc:1035] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862aee2e55284192955041c46f55f8ea" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 32917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:31.600262 25307 mvcc.cc:205] Tried to move back new op lower bound from 7316876212592480256 to 7316876212508426240. Current Snapshot: MvccSnapshot[applied={T|T < 7316876212592480256}]
W20260810 07:25:31.600436 25018 tablet.cc:2369] T 4084eb94554d43bfb0230a39e8621cc7 P 313322cdeb7048778cb6dedb8a4aa76e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:25:31.614008 23185 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:31.614166 23185 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:31.618844 23185 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eebcdb40543d45c5aef6817f84cef282: Sending DeleteTablet for 3 replicas of tablet 4084eb94554d43bfb0230a39e8621cc7
I20260810 07:25:31.619800 24901 tablet_service.cc:1514] Processing DeleteTablet for tablet 4084eb94554d43bfb0230a39e8621cc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:31 UTC) from {username='oryx', principal='oryx/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:60380
I20260810 07:25:31.619913 25130 tablet_service.cc:1514] Processing DeleteTablet for tablet 4084eb94554d43bfb0230a39e8621cc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:31 UTC) from {username='oryx', principal='oryx/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:56216
I20260810 07:25:31.620574 25322 tablet_replica.cc:330] T 4084eb94554d43bfb0230a39e8621cc7 P 862aee2e55284192955041c46f55f8ea: stopping tablet replica
I20260810 07:25:31.620537 24659 tablet_service.cc:1514] Processing DeleteTablet for tablet 4084eb94554d43bfb0230a39e8621cc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:31 UTC) from {username='oryx', principal='oryx/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:60778
I20260810 07:25:31.621052 25323 tablet_replica.cc:330] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91: stopping tablet replica
I20260810 07:25:31.621454 25322 raft_consensus.cc:2229] T 4084eb94554d43bfb0230a39e8621cc7 P 862aee2e55284192955041c46f55f8ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:25:31.621793 25322 raft_consensus.cc:2258] T 4084eb94554d43bfb0230a39e8621cc7 P 862aee2e55284192955041c46f55f8ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:31.622033 25321 tablet_replica.cc:330] T 4084eb94554d43bfb0230a39e8621cc7 P 313322cdeb7048778cb6dedb8a4aa76e: stopping tablet replica
I20260810 07:25:31.622252 25323 raft_consensus.cc:2229] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:25:31.622380 25321 raft_consensus.cc:2229] T 4084eb94554d43bfb0230a39e8621cc7 P 313322cdeb7048778cb6dedb8a4aa76e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:25:31.622701 25321 raft_consensus.cc:2258] T 4084eb94554d43bfb0230a39e8621cc7 P 313322cdeb7048778cb6dedb8a4aa76e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:31.622736 25323 raft_consensus.cc:2258] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:31.623601 25322 ts_tablet_manager.cc:1905] T 4084eb94554d43bfb0230a39e8621cc7 P 862aee2e55284192955041c46f55f8ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:31.624162 25323 ts_tablet_manager.cc:1905] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:31.624389 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 23137
I20260810 07:25:31.629640 25323 ts_tablet_manager.cc:1918] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:25:31.629820 25323 log.cc:1196] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-0/wal/wals/4084eb94554d43bfb0230a39e8621cc7
I20260810 07:25:31.630329 25323 ts_tablet_manager.cc:1939] T 4084eb94554d43bfb0230a39e8621cc7 P 373169c708e84b2c9a7e52676b00de91: Deleting consensus metadata
I20260810 07:25:31.631438 25322 ts_tablet_manager.cc:1918] T 4084eb94554d43bfb0230a39e8621cc7 P 862aee2e55284192955041c46f55f8ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:25:31.631554 25322 log.cc:1196] T 4084eb94554d43bfb0230a39e8621cc7 P 862aee2e55284192955041c46f55f8ea: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/wal/wals/4084eb94554d43bfb0230a39e8621cc7
I20260810 07:25:31.631873 25322 ts_tablet_manager.cc:1939] T 4084eb94554d43bfb0230a39e8621cc7 P 862aee2e55284192955041c46f55f8ea: Deleting consensus metadata
W20260810 07:25:31.640872 24565 connection.cc:535] server connection from 127.0.0.1:60778 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 07:25:31.640887 25051 connection.cc:535] server connection from 127.0.0.1:56216 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 07:25:31.641305 25321 ts_tablet_manager.cc:1905] T 4084eb94554d43bfb0230a39e8621cc7 P 313322cdeb7048778cb6dedb8a4aa76e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:31.644217 25321 ts_tablet_manager.cc:1918] T 4084eb94554d43bfb0230a39e8621cc7 P 313322cdeb7048778cb6dedb8a4aa76e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:25:31.644345 25321 log.cc:1196] T 4084eb94554d43bfb0230a39e8621cc7 P 313322cdeb7048778cb6dedb8a4aa76e: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/wal/wals/4084eb94554d43bfb0230a39e8621cc7
I20260810 07:25:31.644696 25321 ts_tablet_manager.cc:1939] T 4084eb94554d43bfb0230a39e8621cc7 P 313322cdeb7048778cb6dedb8a4aa76e: Deleting consensus metadata
I20260810 07:25:31.645802 24347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.623567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56474 (local address 127.12.157.253:33175)
0810 07:25:31.623810 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:31.623815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:31.633275 (+ 9460us) server_negotiation.cc:407] Connection header received
0810 07:25:31.633437 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:31.633440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:31.633509 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:31.633601 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:31.634340 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.635014 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.635958 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.636241 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.636303 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.642260 (+ 5957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:31.642264 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:31.642267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:31.642293 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:31.643775 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:31.644245 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:31.644253 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:31.644255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:31.644367 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:31.644679 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:31.644683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:31.644685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:31.644931 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:31.645112 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:31.645487 (+ 375us) server_negotiation.cc:299] Negotiation successful
0810 07:25:31.645614 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
W20260810 07:25:31.647902 24815 connection.cc:441] server connection from 127.0.0.1:60380 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60380 (request call id 1) could send its response
I20260810 07:25:31.654767 25258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.623683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32902 (local address 127.12.157.252:39867)
0810 07:25:31.623916 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:31.623920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:31.645979 (+ 22059us) server_negotiation.cc:407] Connection header received
0810 07:25:31.646184 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:31.646189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:31.646250 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:31.646312 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:31.647189 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.647723 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.648638 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.649095 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.649213 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.650617 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:31.650621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:31.650624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:31.650651 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:31.652514 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:31.653066 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:31.653070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:31.653073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:31.653164 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:31.653451 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:31.653454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:31.653456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:31.653650 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:31.653773 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:31.654477 (+ 704us) server_negotiation.cc:299] Negotiation successful
0810 07:25:31.654604 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260810 07:25:31.660765 24347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.650586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56490 (local address 127.12.157.253:33175)
0810 07:25:31.650695 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:31.650699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:31.651094 (+ 395us) server_negotiation.cc:407] Connection header received
0810 07:25:31.651291 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:31.651296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:31.651351 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:31.651403 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:31.652846 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.653625 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.654808 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.655082 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.655141 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.656916 (+ 1775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:31.656920 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:31.656923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:31.656976 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:31.658957 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:31.659452 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:31.659456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:31.659458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:31.659550 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:31.659852 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:31.659856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:31.659858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:31.660086 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:31.660228 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:31.660469 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 07:25:31.660579 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 07:25:31.661075 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:31.651180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32914 (local address 127.12.157.252:39867)
0810 07:25:31.651297 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:31.651301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:31.651561 (+ 260us) server_negotiation.cc:407] Connection header received
0810 07:25:31.651731 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:31.651734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:31.651787 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:31.651837 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:31.652692 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.653508 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.654707 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:31.655054 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:31.655124 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:31.657462 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:31.657465 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:31.657468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:31.657499 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:31.659234 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:31.659712 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:31.659716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:31.659717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:31.659808 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:31.660112 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:31.660115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:31.660117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:31.660315 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:31.660414 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:31.660763 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 07:25:31.660889 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 07:25:32.605733 25335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:32.605213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.12.157.194:37919)
0810 07:25:32.605504 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:32.605612 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":113,"threads_started":1}
W20260810 07:25:32.606217 25015 heartbeater.cc:644] Failed to heartbeat to 127.12.157.254:46027 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
I20260810 07:25:32.608333 25336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:32.607943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.12.157.193:41013)
0810 07:25:32.608179 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:32.608253 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":78,"threads_started":1}
W20260810 07:25:32.608662 24780 heartbeater.cc:644] Failed to heartbeat to 127.12.157.254:46027 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
I20260810 07:25:32.609742 25339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:32.609197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.12.157.195:47671)
0810 07:25:32.609561 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:32.609650 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":80,"threads_started":1}
W20260810 07:25:32.610213 25238 heartbeater.cc:644] Failed to heartbeat to 127.12.157.254:46027 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
I20260810 07:25:33.119122 25341 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eebcdb40543d45c5aef6817f84cef282)
I20260810 07:25:33.119407 25341 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: false } }
I20260810 07:25:33.119508 25343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eebcdb40543d45c5aef6817f84cef282)
I20260810 07:25:33.119715 25343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: false } }
I20260810 07:25:33.120456 25341 leader_election.cc:288] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eebcdb40543d45c5aef6817f84cef282 (127.12.157.254:46027), 1115ade1c930472ea39553bfd9614863 (127.12.157.252:39867)
I20260810 07:25:33.121222 25346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:33.120593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.0.0.1:53052)
0810 07:25:33.121010 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:33.121105 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":78,"threads_started":1}
I20260810 07:25:33.120918 25343 leader_election.cc:288] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eebcdb40543d45c5aef6817f84cef282 (127.12.157.254:46027), 7f2fdbbd52b1447a8119c5d0ba547dbf (127.12.157.253:33175)
W20260810 07:25:33.121414 24124 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
I20260810 07:25:33.121999 25347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:33.120880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.0.0.1:53062)
0810 07:25:33.121817 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:33.121890 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":881,"thread_start_us":86,"threads_started":1}
W20260810 07:25:33.122160 24429 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
I20260810 07:25:33.122730 25346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:33.122480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.0.0.1:53064)
0810 07:25:33.122623 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:33.122668 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260810 07:25:33.122921 24124 leader_election.cc:334] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer eebcdb40543d45c5aef6817f84cef282 (127.12.157.254:46027): Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
I20260810 07:25:33.123111 25347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:33.122872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.0.0.1:53078)
0810 07:25:33.123001 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:33.123049 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260810 07:25:33.123245 24429 leader_election.cc:334] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer eebcdb40543d45c5aef6817f84cef282 (127.12.157.254:46027): Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
I20260810 07:25:33.125380 25345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:33.120652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:39867 (local address 127.0.0.1:32928)
0810 07:25:33.121145 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:33.121162 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:33.121270 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:33.121722 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:33.121728 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:33.121750 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:33.121887 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:33.121894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:33.122954 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:33.122958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:33.124184 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:33.124195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:33.124898 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:33.124901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:33.125090 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:33.125111 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 07:25:33.125208 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 07:25:33.125274 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":22,"thread_start_us":60,"threads_started":1}
I20260810 07:25:33.125452 25350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:33.121123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32928 (local address 127.12.157.252:39867)
0810 07:25:33.121456 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:33.121460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:33.121475 (+ 15us) server_negotiation.cc:407] Connection header received
0810 07:25:33.121525 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:33.121527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:33.121572 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:33.121712 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:33.122017 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:33.122800 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:33.124315 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:33.124754 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:33.124921 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:33.125005 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:33.125246 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 07:25:33.125321 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":247,"thread_start_us":65,"threads_started":1}
I20260810 07:25:33.125931 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1115ade1c930472ea39553bfd9614863" is_pre_election: true
I20260810 07:25:33.126171 25348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:33.120751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:33175 (local address 127.0.0.1:56496)
0810 07:25:33.121535 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:33.121550 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:33.121652 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:33.122556 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:33.122558 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:33.122578 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:33.122672 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:33.122678 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:33.123643 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:33.123647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:33.124717 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:33.124727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:33.125792 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:33.125795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:33.125878 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:33.125899 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 07:25:33.125982 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 07:25:33.126056 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":2,"thread_start_us":55,"threads_started":1}
I20260810 07:25:33.126154 24473 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f2fdbbd52b1447a8119c5d0ba547dbf in term 1.
I20260810 07:25:33.126277 25349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:33.120903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56496 (local address 127.12.157.253:33175)
0810 07:25:33.121463 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:33.121468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:33.121628 (+ 160us) server_negotiation.cc:407] Connection header received
0810 07:25:33.122022 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:33.122025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:33.122211 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:33.122298 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:33.122808 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:33.123529 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:33.124846 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:33.125560 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:33.125676 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:33.125715 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:33.126025 (+ 310us) server_negotiation.cc:299] Negotiation successful
0810 07:25:33.126135 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":493,"thread_start_us":60,"threads_started":1}
I20260810 07:25:33.126601 24131 leader_election.cc:302] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1115ade1c930472ea39553bfd9614863, 7f2fdbbd52b1447a8119c5d0ba547dbf; no voters: eebcdb40543d45c5aef6817f84cef282
I20260810 07:25:33.126678 24162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1115ade1c930472ea39553bfd9614863" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" is_pre_election: true
I20260810 07:25:33.126880 24162 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1115ade1c930472ea39553bfd9614863 in term 1.
I20260810 07:25:33.126956 25341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 07:25:33.127009 25341 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [term 1 FOLLOWER]: Starting leader election (detected failure of leader eebcdb40543d45c5aef6817f84cef282)
I20260810 07:25:33.127065 25341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [term 1 FOLLOWER]: Advancing to term 2
I20260810 07:25:33.127260 24431 leader_election.cc:302] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1115ade1c930472ea39553bfd9614863, 7f2fdbbd52b1447a8119c5d0ba547dbf; no voters: eebcdb40543d45c5aef6817f84cef282
I20260810 07:25:33.127499 25343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 07:25:33.127573 25343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [term 1 FOLLOWER]: Starting leader election (detected failure of leader eebcdb40543d45c5aef6817f84cef282)
I20260810 07:25:33.127635 25343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [term 1 FOLLOWER]: Advancing to term 2
I20260810 07:25:33.128222 25341 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: false } }
I20260810 07:25:33.128592 25341 leader_election.cc:288] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [CANDIDATE]: Term 2 election: Requested vote from peers eebcdb40543d45c5aef6817f84cef282 (127.12.157.254:46027), 1115ade1c930472ea39553bfd9614863 (127.12.157.252:39867)
I20260810 07:25:33.128808 25343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: false } }
I20260810 07:25:33.128926 25345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:33.128679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.0.0.1:53082)
0810 07:25:33.128811 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:33.128863 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260810 07:25:33.129019 24473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1115ade1c930472ea39553bfd9614863"
I20260810 07:25:33.129187 24473 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f2fdbbd52b1447a8119c5d0ba547dbf in current term 2: Already voted for candidate 1115ade1c930472ea39553bfd9614863 in this term.
I20260810 07:25:33.129245 25343 leader_election.cc:288] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [CANDIDATE]: Term 2 election: Requested vote from peers eebcdb40543d45c5aef6817f84cef282 (127.12.157.254:46027), 7f2fdbbd52b1447a8119c5d0ba547dbf (127.12.157.253:33175)
I20260810 07:25:33.129567 25348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:33.129327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.0.0.1:53094)
0810 07:25:33.129432 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:33.129501 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260810 07:25:33.129820 25345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:33.129643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.0.0.1:53100)
0810 07:25:33.129734 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:33.129776 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 07:25:33.129936 24162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1115ade1c930472ea39553bfd9614863" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf"
I20260810 07:25:33.130126 24162 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1115ade1c930472ea39553bfd9614863 in current term 2: Already voted for candidate 7f2fdbbd52b1447a8119c5d0ba547dbf in this term.
W20260810 07:25:33.130292 24124 leader_election.cc:334] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer eebcdb40543d45c5aef6817f84cef282 (127.12.157.254:46027): Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
I20260810 07:25:33.130357 25348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:33.130135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.0.0.1:53112)
0810 07:25:33.130235 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:33.130294 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 07:25:33.130388 24124 leader_election.cc:302] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f2fdbbd52b1447a8119c5d0ba547dbf; no voters: 1115ade1c930472ea39553bfd9614863, eebcdb40543d45c5aef6817f84cef282
W20260810 07:25:33.130484 24429 leader_election.cc:334] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer eebcdb40543d45c5aef6817f84cef282 (127.12.157.254:46027): Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
I20260810 07:25:33.130519 25341 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 07:25:33.130548 24429 leader_election.cc:302] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1115ade1c930472ea39553bfd9614863; no voters: 7f2fdbbd52b1447a8119c5d0ba547dbf, eebcdb40543d45c5aef6817f84cef282
I20260810 07:25:33.130671 25343 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 07:25:33.607389 25357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:33.606854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.12.157.194:56151)
0810 07:25:33.607187 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:33.607267 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":130,"threads_started":1}
I20260810 07:25:33.609663 25360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:33.609277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.12.157.193:48201)
0810 07:25:33.609526 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:33.609581 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":81,"threads_started":1}
I20260810 07:25:33.611174 25362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:33.610725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.12.157.195:49315)
0810 07:25:33.610999 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:33.611065 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":94,"threads_started":1}
I20260810 07:25:34.608795 25363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.608245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.12.157.194:45201)
0810 07:25:34.608603 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.608678 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":100,"threads_started":1}
I20260810 07:25:34.610844 25364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.610335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.12.157.193:38579)
0810 07:25:34.610677 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.610733 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":106,"threads_started":1}
I20260810 07:25:34.612154 25365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.611799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.12.157.195:55249)
0810 07:25:34.612027 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.612077 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":70,"threads_started":1}
I20260810 07:25:34.636839 25366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:25:34.637101 25366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: false } }
I20260810 07:25:34.637667 25366 leader_election.cc:288] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers eebcdb40543d45c5aef6817f84cef282 (127.12.157.254:46027), 7f2fdbbd52b1447a8119c5d0ba547dbf (127.12.157.253:33175)
I20260810 07:25:34.638160 24162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1115ade1c930472ea39553bfd9614863" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" is_pre_election: true
I20260810 07:25:34.638341 24162 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1115ade1c930472ea39553bfd9614863 in term 2.
I20260810 07:25:34.638362 25367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.637894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.0.0.1:53330)
0810 07:25:34.638141 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.638201 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":95,"threads_started":1}
I20260810 07:25:34.638715 24431 leader_election.cc:302] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1115ade1c930472ea39553bfd9614863, 7f2fdbbd52b1447a8119c5d0ba547dbf; no voters:
I20260810 07:25:34.638904 25366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 07:25:34.638988 25366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:25:34.639043 25366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [term 2 FOLLOWER]: Advancing to term 3
I20260810 07:25:34.639314 25367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.639076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.0.0.1:53344)
0810 07:25:34.639213 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.639255 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260810 07:25:34.639439 24429 leader_election.cc:334] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer eebcdb40543d45c5aef6817f84cef282 (127.12.157.254:46027): Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
I20260810 07:25:34.640465 25366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: false } }
I20260810 07:25:34.641117 25366 leader_election.cc:288] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [CANDIDATE]: Term 3 election: Requested vote from peers eebcdb40543d45c5aef6817f84cef282 (127.12.157.254:46027), 7f2fdbbd52b1447a8119c5d0ba547dbf (127.12.157.253:33175)
I20260810 07:25:34.641264 25367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.640909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.0.0.1:53358)
0810 07:25:34.641133 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.641200 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260810 07:25:34.641433 24162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1115ade1c930472ea39553bfd9614863" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf"
I20260810 07:25:34.641568 24162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [term 2 FOLLOWER]: Advancing to term 3
I20260810 07:25:34.641858 25367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.641672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.0.0.1:53374)
0810 07:25:34.641755 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.641799 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260810 07:25:34.641963 24429 leader_election.cc:334] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer eebcdb40543d45c5aef6817f84cef282 (127.12.157.254:46027): Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
I20260810 07:25:34.642598 24162 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1115ade1c930472ea39553bfd9614863 in term 3.
I20260810 07:25:34.642894 24431 leader_election.cc:302] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1115ade1c930472ea39553bfd9614863, 7f2fdbbd52b1447a8119c5d0ba547dbf; no voters: eebcdb40543d45c5aef6817f84cef282
I20260810 07:25:34.643069 25366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 07:25:34.643266 25366 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [term 3 LEADER]: Becoming Leader. State: Replica: 1115ade1c930472ea39553bfd9614863, State: Running, Role: LEADER
I20260810 07:25:34.643543 25366 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: false } }
I20260810 07:25:34.644711 25369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1115ade1c930472ea39553bfd9614863. Latest consensus state: current_term: 3 leader_uuid: "1115ade1c930472ea39553bfd9614863" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: false } } }
I20260810 07:25:34.644804 25369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:34.645125 25371 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:25:34.647408 25371 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:25:34.647835 25371 catalog_manager.cc:1259] Loaded cluster ID: 45c8a83eed7741dfa445f32c118347c1
I20260810 07:25:34.647882 25371 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:25:34.648495 25371 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:25:34.648911 25371 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863: Loaded TSK: 0
I20260810 07:25:34.649914 25371 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:25:34.720278 24451 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:25:34.723572 24451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:25:34.726778 25367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.726480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.0.0.1:53388)
0810 07:25:34.726628 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.726697 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 07:25:34.726859 24162 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [term 3 FOLLOWER]: Refusing update from remote peer 1115ade1c930472ea39553bfd9614863: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260810 07:25:34.727380 25369 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260810 07:25:34.727581 25367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.727336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.0.0.1:53392)
0810 07:25:34.727450 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.727496 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 07:25:34.727761 24429 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 -> Peer eebcdb40543d45c5aef6817f84cef282 (127.12.157.254:46027): Couldn't send request to peer eebcdb40543d45c5aef6817f84cef282. Status: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 07:25:34.728935 25373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1115ade1c930472ea39553bfd9614863. Latest consensus state: current_term: 3 leader_uuid: "1115ade1c930472ea39553bfd9614863" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: false } } }
I20260810 07:25:34.729102 25373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:25:34.730203 25366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "1115ade1c930472ea39553bfd9614863" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: false } } }
I20260810 07:25:34.730316 25366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:34.730626 25373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "1115ade1c930472ea39553bfd9614863" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "eebcdb40543d45c5aef6817f84cef282" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46027 } } peers { permanent_uuid: "7f2fdbbd52b1447a8119c5d0ba547dbf" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33175 } attrs { promote: false } } peers { permanent_uuid: "1115ade1c930472ea39553bfd9614863" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 39867 } attrs { promote: false } } }
I20260810 07:25:34.730741 25373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f2fdbbd52b1447a8119c5d0ba547dbf [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:25:34.731326 25374 mvcc.cc:205] Tried to move back new op lower bound from 7316876225436004352 to 7316876225102708736. Current Snapshot: MvccSnapshot[applied={T|T < 7316876225436004352 or (T in {7316876225436004352})}]
I20260810 07:25:34.740648 25367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.736295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:36441 (local address 127.0.0.1:60798)
0810 07:25:34.736480 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.736497 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:34.736595 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:34.737116 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:34.737120 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:34.737141 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:34.737252 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:34.737259 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:34.738370 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:34.738375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:34.739254 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:34.739264 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:34.740227 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:34.740230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:34.740318 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:34.740347 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 07:25:34.740395 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 07:25:34.740499 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260810 07:25:34.741331 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.736408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60798 (local address 127.12.157.193:36441)
0810 07:25:34.736678 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:34.736684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:34.736703 (+ 19us) server_negotiation.cc:407] Connection header received
0810 07:25:34.736764 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:34.736770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:34.736975 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:34.737373 (+ 398us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:34.737495 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:34.738274 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:34.739381 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:34.740091 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:34.741011 (+ 920us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:34.741067 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:34.741105 (+ 38us) server_negotiation.cc:299] Negotiation successful
0810 07:25:34.741181 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":79,"threads_started":1}
I20260810 07:25:34.741808 25386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.736860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:32917 (local address 127.0.0.1:56222)
0810 07:25:34.737365 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.737381 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:34.737474 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:34.737890 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:34.737893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:34.737913 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:34.738022 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:34.738027 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:34.739121 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:34.739125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:34.740401 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:34.740408 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:34.741411 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:34.741417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:34.741514 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:34.741538 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 07:25:34.741620 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 07:25:34.741687 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":36,"thread_start_us":153,"threads_started":1}
I20260810 07:25:34.741858 24659 tablet_service.cc:1467] Processing CreateTablet for tablet cfc1ca9194444ec1a7becbd0c3b8f054 (DEFAULT_TABLE table=test-table [id=2f50b8eacc6548d29ae4a79595900cb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:34.742053 25389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.736884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:37871 (local address 127.0.0.1:60384)
0810 07:25:34.737691 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.737705 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:34.737785 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:34.738187 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:34.738190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:34.738208 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:34.738286 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:34.738291 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:34.739326 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:34.739330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:34.740463 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:34.740473 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:34.741611 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:34.741616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:34.741701 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:34.741723 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 07:25:34.741794 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 07:25:34.741854 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":19,"thread_start_us":146,"threads_started":1}
I20260810 07:25:34.742161 24659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc1ca9194444ec1a7becbd0c3b8f054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:34.742422 25392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.737226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56222 (local address 127.12.157.195:32917)
0810 07:25:34.737565 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:34.737570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:34.737586 (+ 16us) server_negotiation.cc:407] Connection header received
0810 07:25:34.737641 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:34.737643 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:34.737702 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:34.737831 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:34.738160 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:34.738995 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:34.740524 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:34.741183 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:34.741294 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:34.741344 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:34.741638 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 07:25:34.741719 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":92,"threads_started":1}
I20260810 07:25:34.742439 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.737081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60384 (local address 127.12.157.194:37871)
0810 07:25:34.737986 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:34.737990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:34.738003 (+ 13us) server_negotiation.cc:407] Connection header received
0810 07:25:34.738052 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:34.738054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:34.738098 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:34.738229 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:34.738436 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:34.739221 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:34.740605 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:34.741341 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:34.741497 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:34.741544 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:34.742210 (+ 666us) server_negotiation.cc:299] Negotiation successful
0810 07:25:34.742312 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":825,"thread_start_us":94,"threads_started":1}
I20260810 07:25:34.743172 25130 tablet_service.cc:1467] Processing CreateTablet for tablet cfc1ca9194444ec1a7becbd0c3b8f054 (DEFAULT_TABLE table=test-table [id=2f50b8eacc6548d29ae4a79595900cb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:34.743459 25130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc1ca9194444ec1a7becbd0c3b8f054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:34.744663 25393 tablet_bootstrap.cc:492] T cfc1ca9194444ec1a7becbd0c3b8f054 P 373169c708e84b2c9a7e52676b00de91: Bootstrap starting.
I20260810 07:25:34.744870 24901 tablet_service.cc:1467] Processing CreateTablet for tablet cfc1ca9194444ec1a7becbd0c3b8f054 (DEFAULT_TABLE table=test-table [id=2f50b8eacc6548d29ae4a79595900cb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:34.745159 24901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc1ca9194444ec1a7becbd0c3b8f054. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:34.745697 25393 tablet_bootstrap.cc:654] T cfc1ca9194444ec1a7becbd0c3b8f054 P 373169c708e84b2c9a7e52676b00de91: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:34.745880 25396 tablet_bootstrap.cc:492] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea: Bootstrap starting.
I20260810 07:25:34.746938 25396 tablet_bootstrap.cc:654] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:34.747686 25402 tablet_bootstrap.cc:492] T cfc1ca9194444ec1a7becbd0c3b8f054 P 313322cdeb7048778cb6dedb8a4aa76e: Bootstrap starting.
I20260810 07:25:34.748339 25396 tablet_bootstrap.cc:492] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea: No bootstrap required, opened a new log
I20260810 07:25:34.748443 25393 tablet_bootstrap.cc:492] T cfc1ca9194444ec1a7becbd0c3b8f054 P 373169c708e84b2c9a7e52676b00de91: No bootstrap required, opened a new log
I20260810 07:25:34.748457 25396 ts_tablet_manager.cc:1397] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 07:25:34.748548 25393 ts_tablet_manager.cc:1397] T cfc1ca9194444ec1a7becbd0c3b8f054 P 373169c708e84b2c9a7e52676b00de91: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 07:25:34.748636 25402 tablet_bootstrap.cc:654] T cfc1ca9194444ec1a7becbd0c3b8f054 P 313322cdeb7048778cb6dedb8a4aa76e: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:34.749019 25396 raft_consensus.cc:348] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "373169c708e84b2c9a7e52676b00de91" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 36441 } } peers { permanent_uuid: "313322cdeb7048778cb6dedb8a4aa76e" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37871 } } peers { permanent_uuid: "862aee2e55284192955041c46f55f8ea" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 32917 } }
I20260810 07:25:34.749157 25396 raft_consensus.cc:374] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:34.749094 25393 raft_consensus.cc:348] T cfc1ca9194444ec1a7becbd0c3b8f054 P 373169c708e84b2c9a7e52676b00de91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "373169c708e84b2c9a7e52676b00de91" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 36441 } } peers { permanent_uuid: "313322cdeb7048778cb6dedb8a4aa76e" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37871 } } peers { permanent_uuid: "862aee2e55284192955041c46f55f8ea" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 32917 } }
I20260810 07:25:34.749199 25396 raft_consensus.cc:729] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862aee2e55284192955041c46f55f8ea, State: Initialized, Role: FOLLOWER
I20260810 07:25:34.749217 25393 raft_consensus.cc:374] T cfc1ca9194444ec1a7becbd0c3b8f054 P 373169c708e84b2c9a7e52676b00de91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:34.749262 25393 raft_consensus.cc:729] T cfc1ca9194444ec1a7becbd0c3b8f054 P 373169c708e84b2c9a7e52676b00de91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 373169c708e84b2c9a7e52676b00de91, State: Initialized, Role: FOLLOWER
I20260810 07:25:34.749341 25396 consensus_queue.cc:260] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "373169c708e84b2c9a7e52676b00de91" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 36441 } } peers { permanent_uuid: "313322cdeb7048778cb6dedb8a4aa76e" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37871 } } peers { permanent_uuid: "862aee2e55284192955041c46f55f8ea" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 32917 } }
I20260810 07:25:34.749392 25393 consensus_queue.cc:260] T cfc1ca9194444ec1a7becbd0c3b8f054 P 373169c708e84b2c9a7e52676b00de91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "373169c708e84b2c9a7e52676b00de91" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 36441 } } peers { permanent_uuid: "313322cdeb7048778cb6dedb8a4aa76e" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37871 } } peers { permanent_uuid: "862aee2e55284192955041c46f55f8ea" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 32917 } }
I20260810 07:25:34.749754 25396 ts_tablet_manager.cc:1428] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 07:25:34.749806 25393 ts_tablet_manager.cc:1428] T cfc1ca9194444ec1a7becbd0c3b8f054 P 373169c708e84b2c9a7e52676b00de91: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 07:25:34.750183 25402 tablet_bootstrap.cc:492] T cfc1ca9194444ec1a7becbd0c3b8f054 P 313322cdeb7048778cb6dedb8a4aa76e: No bootstrap required, opened a new log
I20260810 07:25:34.750262 25402 ts_tablet_manager.cc:1397] T cfc1ca9194444ec1a7becbd0c3b8f054 P 313322cdeb7048778cb6dedb8a4aa76e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 07:25:34.750733 25364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.750262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.12.157.193:38031)
0810 07:25:34.750603 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.750670 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252}
I20260810 07:25:34.750702 25402 raft_consensus.cc:348] T cfc1ca9194444ec1a7becbd0c3b8f054 P 313322cdeb7048778cb6dedb8a4aa76e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "373169c708e84b2c9a7e52676b00de91" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 36441 } } peers { permanent_uuid: "313322cdeb7048778cb6dedb8a4aa76e" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37871 } } peers { permanent_uuid: "862aee2e55284192955041c46f55f8ea" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 32917 } }
I20260810 07:25:34.750804 25402 raft_consensus.cc:374] T cfc1ca9194444ec1a7becbd0c3b8f054 P 313322cdeb7048778cb6dedb8a4aa76e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:34.750855 25402 raft_consensus.cc:729] T cfc1ca9194444ec1a7becbd0c3b8f054 P 313322cdeb7048778cb6dedb8a4aa76e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 313322cdeb7048778cb6dedb8a4aa76e, State: Initialized, Role: FOLLOWER
I20260810 07:25:34.750984 25402 consensus_queue.cc:260] T cfc1ca9194444ec1a7becbd0c3b8f054 P 313322cdeb7048778cb6dedb8a4aa76e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "373169c708e84b2c9a7e52676b00de91" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 36441 } } peers { permanent_uuid: "313322cdeb7048778cb6dedb8a4aa76e" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37871 } } peers { permanent_uuid: "862aee2e55284192955041c46f55f8ea" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 32917 } }
I20260810 07:25:34.751333 25402 ts_tablet_manager.cc:1428] T cfc1ca9194444ec1a7becbd0c3b8f054 P 313322cdeb7048778cb6dedb8a4aa76e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:34.751504 25364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.751273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.12.157.193:46801)
0810 07:25:34.751385 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.751443 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 07:25:34.753484 25363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.753220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.12.157.194:49843)
0810 07:25:34.753343 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.753401 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 07:25:34.754241 24779 heartbeater.cc:503] Master 127.12.157.252:39867 was elected leader, sending a full tablet report...
I20260810 07:25:34.755468 25365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.755095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.12.157.195:44749)
0810 07:25:34.755241 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.755310 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 07:25:34.756171 25363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.755951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.12.157.194:36849)
0810 07:25:34.756070 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.756111 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 07:25:34.756915 25365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.756701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.12.157.195:44187)
0810 07:25:34.756822 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.756862 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260810 07:25:34.757786 25014 heartbeater.cc:503] Master 127.12.157.252:39867 was elected leader, sending a full tablet report...
I20260810 07:25:34.758888 25236 heartbeater.cc:503] Master 127.12.157.252:39867 was elected leader, sending a full tablet report...
I20260810 07:25:34.865960 25407 raft_consensus.cc:482] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:25:34.866164 25407 raft_consensus.cc:504] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "373169c708e84b2c9a7e52676b00de91" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 36441 } } peers { permanent_uuid: "313322cdeb7048778cb6dedb8a4aa76e" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37871 } } peers { permanent_uuid: "862aee2e55284192955041c46f55f8ea" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 32917 } }
I20260810 07:25:34.867475 25407 leader_election.cc:288] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 373169c708e84b2c9a7e52676b00de91 (127.12.157.193:36441), 313322cdeb7048778cb6dedb8a4aa76e (127.12.157.194:37871)
I20260810 07:25:34.872200 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.868115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:52953 (local address 127.12.157.194:37871)
0810 07:25:34.868248 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:34.868253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:34.868367 (+ 114us) server_negotiation.cc:407] Connection header received
0810 07:25:34.868526 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:34.868530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:34.868578 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:34.868654 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:34.869037 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:34.869748 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:34.871078 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:34.871545 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:34.871702 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:34.871757 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:34.871979 (+ 222us) server_negotiation.cc:299] Negotiation successful
0810 07:25:34.872062 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 07:25:34.872251 25365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.867708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:36441 (local address 127.12.157.195:36551)
0810 07:25:34.868130 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.868144 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:34.868260 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:34.868534 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:34.868537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:34.868549 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:34.868677 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:34.868682 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:34.869535 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:34.869540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:34.870771 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:34.870782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:34.871815 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:34.871819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:34.871934 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:34.871958 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 07:25:34.872024 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 07:25:34.872095 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":1}
I20260810 07:25:34.872295 25412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.867759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:37871 (local address 127.12.157.195:52953)
0810 07:25:34.868267 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.868282 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:34.868405 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:34.868779 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:34.868782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:34.868800 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:34.868906 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:34.868912 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:34.869870 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:34.869873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:34.870961 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:34.870968 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:34.871702 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:34.871706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:34.871816 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:34.871847 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 07:25:34.871957 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 07:25:34.872037 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":128,"threads_started":1}
I20260810 07:25:34.872819 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.867823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:36551 (local address 127.12.157.193:36441)
0810 07:25:34.867997 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:34.868001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:34.868289 (+ 288us) server_negotiation.cc:407] Connection header received
0810 07:25:34.868351 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:34.868357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:34.868413 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:34.868514 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:34.868792 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:34.869420 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:34.870908 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:34.871574 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:34.871686 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:34.871752 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:34.872609 (+ 857us) server_negotiation.cc:299] Negotiation successful
0810 07:25:34.872692 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 07:25:34.872802 24932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfc1ca9194444ec1a7becbd0c3b8f054" candidate_uuid: "862aee2e55284192955041c46f55f8ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "313322cdeb7048778cb6dedb8a4aa76e" is_pre_election: true
I20260810 07:25:34.873018 24932 raft_consensus.cc:2454] T cfc1ca9194444ec1a7becbd0c3b8f054 P 313322cdeb7048778cb6dedb8a4aa76e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862aee2e55284192955041c46f55f8ea in term 0.
I20260810 07:25:34.873279 24700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfc1ca9194444ec1a7becbd0c3b8f054" candidate_uuid: "862aee2e55284192955041c46f55f8ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "373169c708e84b2c9a7e52676b00de91" is_pre_election: true
I20260810 07:25:34.873502 24700 raft_consensus.cc:2454] T cfc1ca9194444ec1a7becbd0c3b8f054 P 373169c708e84b2c9a7e52676b00de91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 862aee2e55284192955041c46f55f8ea in term 0.
I20260810 07:25:34.873519 25051 leader_election.cc:302] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 313322cdeb7048778cb6dedb8a4aa76e, 862aee2e55284192955041c46f55f8ea; no voters:
I20260810 07:25:34.873821 25407 raft_consensus.cc:2781] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:25:34.873890 25407 raft_consensus.cc:482] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260810 07:25:34.873864 24782 tablet.cc:2369] T cfc1ca9194444ec1a7becbd0c3b8f054 P 373169c708e84b2c9a7e52676b00de91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:25:34.873930 25407 raft_consensus.cc:3037] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:34.875075 25407 raft_consensus.cc:504] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "373169c708e84b2c9a7e52676b00de91" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 36441 } } peers { permanent_uuid: "313322cdeb7048778cb6dedb8a4aa76e" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37871 } } peers { permanent_uuid: "862aee2e55284192955041c46f55f8ea" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 32917 } }
I20260810 07:25:34.875468 25407 leader_election.cc:288] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea [CANDIDATE]: Term 1 election: Requested vote from peers 373169c708e84b2c9a7e52676b00de91 (127.12.157.193:36441), 313322cdeb7048778cb6dedb8a4aa76e (127.12.157.194:37871)
I20260810 07:25:34.875821 24700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfc1ca9194444ec1a7becbd0c3b8f054" candidate_uuid: "862aee2e55284192955041c46f55f8ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "373169c708e84b2c9a7e52676b00de91"
I20260810 07:25:34.875826 24932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfc1ca9194444ec1a7becbd0c3b8f054" candidate_uuid: "862aee2e55284192955041c46f55f8ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "313322cdeb7048778cb6dedb8a4aa76e"
I20260810 07:25:34.875945 24932 raft_consensus.cc:3037] T cfc1ca9194444ec1a7becbd0c3b8f054 P 313322cdeb7048778cb6dedb8a4aa76e [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:34.875960 24700 raft_consensus.cc:3037] T cfc1ca9194444ec1a7becbd0c3b8f054 P 373169c708e84b2c9a7e52676b00de91 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:34.876868 24700 raft_consensus.cc:2454] T cfc1ca9194444ec1a7becbd0c3b8f054 P 373169c708e84b2c9a7e52676b00de91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862aee2e55284192955041c46f55f8ea in term 1.
I20260810 07:25:34.877118 24932 raft_consensus.cc:2454] T cfc1ca9194444ec1a7becbd0c3b8f054 P 313322cdeb7048778cb6dedb8a4aa76e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 862aee2e55284192955041c46f55f8ea in term 1.
I20260810 07:25:34.877233 25054 leader_election.cc:302] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 373169c708e84b2c9a7e52676b00de91, 862aee2e55284192955041c46f55f8ea; no voters:
I20260810 07:25:34.877463 25407 raft_consensus.cc:2781] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:34.877662 25407 raft_consensus.cc:686] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea [term 1 LEADER]: Becoming Leader. State: Replica: 862aee2e55284192955041c46f55f8ea, State: Running, Role: LEADER
I20260810 07:25:34.878012 25407 consensus_queue.cc:237] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "373169c708e84b2c9a7e52676b00de91" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 36441 } } peers { permanent_uuid: "313322cdeb7048778cb6dedb8a4aa76e" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37871 } } peers { permanent_uuid: "862aee2e55284192955041c46f55f8ea" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 32917 } }
I20260810 07:25:34.878358 25412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.878119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.12.157.195:41105)
0810 07:25:34.878243 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.878303 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 07:25:34.880503 24448 catalog_manager.cc:5495] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 862aee2e55284192955041c46f55f8ea (127.12.157.195). New cstate: current_term: 1 leader_uuid: "862aee2e55284192955041c46f55f8ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "373169c708e84b2c9a7e52676b00de91" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 36441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "313322cdeb7048778cb6dedb8a4aa76e" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862aee2e55284192955041c46f55f8ea" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 32917 } health_report { overall_health: HEALTHY } } }
I20260810 07:25:34.900517 25392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.897113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56238 (local address 127.12.157.195:32917)
0810 07:25:34.897259 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:34.897263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:34.897277 (+ 14us) server_negotiation.cc:407] Connection header received
0810 07:25:34.897421 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:34.897424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:34.897479 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:34.897567 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:25:34.897826 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:34.898396 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:34.899197 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:34.899406 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:34.899492 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:34.899718 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:25:34.900139 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:25:34.900227 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:34.900286 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 07:25:34.900333 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 07:25:34.903456 24932 raft_consensus.cc:1261] T cfc1ca9194444ec1a7becbd0c3b8f054 P 313322cdeb7048778cb6dedb8a4aa76e [term 1 FOLLOWER]: Refusing update from remote peer 862aee2e55284192955041c46f55f8ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:34.903590 24700 raft_consensus.cc:1261] T cfc1ca9194444ec1a7becbd0c3b8f054 P 373169c708e84b2c9a7e52676b00de91 [term 1 FOLLOWER]: Refusing update from remote peer 862aee2e55284192955041c46f55f8ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:34.903988 25407 consensus_queue.cc:1035] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "313322cdeb7048778cb6dedb8a4aa76e" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:34.904187 25413 consensus_queue.cc:1035] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "373169c708e84b2c9a7e52676b00de91" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:34.906885 25363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.906588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.12.157.194:43011)
0810 07:25:34.906739 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.906802 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260810 07:25:34.908617 25364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.908138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.12.157.193:47055)
0810 07:25:34.908464 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.908536 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245}
I20260810 07:25:34.909945 25419 mvcc.cc:205] Tried to move back new op lower bound from 7316876226159026176 to 7316876226062901248. Current Snapshot: MvccSnapshot[applied={T|T < 7316876226159026176}]
I20260810 07:25:34.910669 25363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.910362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.12.157.194:43507)
0810 07:25:34.910547 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.910600 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260810 07:25:34.910943 25364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.910672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.12.157.193:33975)
0810 07:25:34.910825 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.910874 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260810 07:25:34.912120 25412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.909017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.12.157.195:52803)
0810 07:25:34.911987 (+ 2970us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.912047 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2897}
I20260810 07:25:34.912825 25365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.912544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46027 (local address 127.12.157.195:49319)
0810 07:25:34.912702 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.912772 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:46027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260810 07:25:34.921288 24448 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:34.921486 24448 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:34.925573 24448 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1115ade1c930472ea39553bfd9614863: Sending DeleteTablet for 3 replicas of tablet cfc1ca9194444ec1a7becbd0c3b8f054
I20260810 07:25:34.926473 24901 tablet_service.cc:1514] Processing DeleteTablet for tablet cfc1ca9194444ec1a7becbd0c3b8f054 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:34 UTC) from {username='oryx', principal='oryx/127.12.157.252@KRBTEST.COM'} at 127.0.0.1:60384
I20260810 07:25:34.926818 24659 tablet_service.cc:1514] Processing DeleteTablet for tablet cfc1ca9194444ec1a7becbd0c3b8f054 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:34 UTC) from {username='oryx', principal='oryx/127.12.157.252@KRBTEST.COM'} at 127.0.0.1:60798
I20260810 07:25:34.927155 25130 tablet_service.cc:1514] Processing DeleteTablet for tablet cfc1ca9194444ec1a7becbd0c3b8f054 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:34 UTC) from {username='oryx', principal='oryx/127.12.157.252@KRBTEST.COM'} at 127.0.0.1:56222
I20260810 07:25:34.927255 25435 tablet_replica.cc:330] T cfc1ca9194444ec1a7becbd0c3b8f054 P 313322cdeb7048778cb6dedb8a4aa76e: stopping tablet replica
I20260810 07:25:34.927508 25436 tablet_replica.cc:330] T cfc1ca9194444ec1a7becbd0c3b8f054 P 373169c708e84b2c9a7e52676b00de91: stopping tablet replica
I20260810 07:25:34.927564 25435 raft_consensus.cc:2229] T cfc1ca9194444ec1a7becbd0c3b8f054 P 313322cdeb7048778cb6dedb8a4aa76e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:25:34.927753 25435 raft_consensus.cc:2258] T cfc1ca9194444ec1a7becbd0c3b8f054 P 313322cdeb7048778cb6dedb8a4aa76e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:34.927977 25437 tablet_replica.cc:330] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea: stopping tablet replica
I20260810 07:25:34.928112 25437 raft_consensus.cc:2229] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:25:34.928452 25437 raft_consensus.cc:2258] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:34.928720 25435 ts_tablet_manager.cc:1905] T cfc1ca9194444ec1a7becbd0c3b8f054 P 313322cdeb7048778cb6dedb8a4aa76e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:34.928822 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 24386
I20260810 07:25:34.929136 25437 ts_tablet_manager.cc:1905] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:34.931690 25437 ts_tablet_manager.cc:1918] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:25:34.931756 25437 log.cc:1196] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-2/wal/wals/cfc1ca9194444ec1a7becbd0c3b8f054
I20260810 07:25:34.932000 25437 ts_tablet_manager.cc:1939] T cfc1ca9194444ec1a7becbd0c3b8f054 P 862aee2e55284192955041c46f55f8ea: Deleting consensus metadata
I20260810 07:25:34.932899 25435 ts_tablet_manager.cc:1918] T cfc1ca9194444ec1a7becbd0c3b8f054 P 313322cdeb7048778cb6dedb8a4aa76e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:25:34.933005 25435 log.cc:1196] T cfc1ca9194444ec1a7becbd0c3b8f054 P 313322cdeb7048778cb6dedb8a4aa76e: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786346694975122-12919-0/minicluster-data/ts-1/wal/wals/cfc1ca9194444ec1a7becbd0c3b8f054
I20260810 07:25:34.933290 25435 ts_tablet_manager.cc:1939] T cfc1ca9194444ec1a7becbd0c3b8f054 P 313322cdeb7048778cb6dedb8a4aa76e: Deleting consensus metadata
I20260810 07:25:34.933558 24430 catalog_manager.cc:4841] TS 862aee2e55284192955041c46f55f8ea (127.12.157.195:32917): tablet cfc1ca9194444ec1a7becbd0c3b8f054 (table test-table [id=2f50b8eacc6548d29ae4a79595900cb9]) successfully deleted
I20260810 07:25:34.934751 24428 catalog_manager.cc:4841] TS 313322cdeb7048778cb6dedb8a4aa76e (127.12.157.194:37871): tablet cfc1ca9194444ec1a7becbd0c3b8f054 (table test-table [id=2f50b8eacc6548d29ae4a79595900cb9]) successfully deleted
I20260810 07:25:34.946260 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 24789
I20260810 07:25:34.946587 25389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.946124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:36441 (local address 127.0.0.1:60804)
0810 07:25:34.946400 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.946476 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.193:36441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260810 07:25:34.946816 24431 catalog_manager.cc:4584] TS 373169c708e84b2c9a7e52676b00de91 (127.12.157.193:36441): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cfc1ca9194444ec1a7becbd0c3b8f054: Network error: Client connection negotiation failed: client connection to 127.12.157.193:36441: connect: Connection refused (error 111)
I20260810 07:25:34.973462 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 25024
I20260810 07:25:34.989843 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 23930
I20260810 07:25:34.999061 25389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.998644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:36441 (local address 127.0.0.1:60812)
0810 07:25:34.998849 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.998937 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.193:36441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260810 07:25:34.999922 25389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:34.999644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:36441 (local address 127.0.0.1:60828)
0810 07:25:34.999775 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:34.999838 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.193:36441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 07:25:35.003268 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 24101
2026-08-10T07:25:35Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7499 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:25:35 dist-test-slave-ffrd krb5kdc[25443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:25:35 dist-test-slave-ffrd krb5kdc[25443](info): set up 2 sockets
Aug 10 07:25:35 dist-test-slave-ffrd krb5kdc[25443](info): commencing operation
krb5kdc: starting...
W20260810 07:25:36.285837 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.245s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:25:36 dist-test-slave-ffrd krb5kdc[25443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:25:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:25:36Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.254.keytab.
Entry for principal oryx/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.254.keytab.
I20260810 07:25:36.414369 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:41615
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.254.keytab
--principal=oryx/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:41615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:36.531917 25470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:36.532197 25470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:36.532239 25470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:36.536355 25470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:25:36.536430 25470 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 07:25:36.536448 25470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:36.536466 25470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:36.536484 25470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:25:36.536499 25470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:25:36.542143 25470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:41615
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.254.keytab
--principal=oryx/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:41615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:36.543350 25470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:36.544467 25470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:36.553552 25484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:36.553586 25481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:36.553598 25480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:36.553812 25470 server_base.cc:1034] running on GCE node
I20260810 07:25:36.554621 25470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:36.555810 25470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:36.557066 25470 hybrid_clock.cc:648] HybridClock initialized: now 1786346736557020 us; error 58 us; skew 500 ppm
Aug 10 07:25:36 dist-test-slave-ffrd krb5kdc[25443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346736, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:36.560117 25470 init.cc:377] Logged in from keytab as oryx/127.12.157.254@KRBTEST.COM (short username oryx)
I20260810 07:25:36.561204 25470 webserver.cc:460] Webserver started at http://127.12.157.254:46871/ using document root <none> and password file <none>
I20260810 07:25:36.561797 25470 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:36.561896 25470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:36.562119 25470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:36.564309 25470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "eb0de2dd430e46078a8686e349202b26"
format_stamp: "Formatted at 2026-08-10 07:25:36 on dist-test-slave-ffrd"
server_key: "4a4480edffd63006b7f0465de7a2ae98"
server_key_iv: "c16399ae68411a53fe79658fee32589f"
server_key_version: "encryptionkey@0"
I20260810 07:25:36.564908 25470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "eb0de2dd430e46078a8686e349202b26"
format_stamp: "Formatted at 2026-08-10 07:25:36 on dist-test-slave-ffrd"
server_key: "4a4480edffd63006b7f0465de7a2ae98"
server_key_iv: "c16399ae68411a53fe79658fee32589f"
server_key_version: "encryptionkey@0"
I20260810 07:25:36.569085 25470 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 07:25:36.571676 25494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:36.573351 25470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:25:36.573510 25470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "eb0de2dd430e46078a8686e349202b26"
format_stamp: "Formatted at 2026-08-10 07:25:36 on dist-test-slave-ffrd"
server_key: "4a4480edffd63006b7f0465de7a2ae98"
server_key_iv: "c16399ae68411a53fe79658fee32589f"
server_key_version: "encryptionkey@0"
I20260810 07:25:36.573645 25470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:36.602437 25470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:36.606118 25470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:36.606353 25470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:36.614917 25470 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:41615
I20260810 07:25:36.614946 25597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:41615 every 8 connection(s)
I20260810 07:25:36.615890 25470 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:25:36.618757 25598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:36.622318 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 25470
I20260810 07:25:36.622457 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:25:36.622761 12919 external_mini_cluster.cc:1428] Setting key 606eaac7d5fc1a2c9dda6c77cd8884b2
I20260810 07:25:36.625088 25598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb0de2dd430e46078a8686e349202b26: Bootstrap starting.
I20260810 07:25:36.627316 25598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb0de2dd430e46078a8686e349202b26: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:36.628104 25598 log.cc:824] T 00000000000000000000000000000000 P eb0de2dd430e46078a8686e349202b26: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:36.630110 25598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb0de2dd430e46078a8686e349202b26: No bootstrap required, opened a new log
Aug 10 07:25:36 dist-test-slave-ffrd krb5kdc[25443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM
I20260810 07:25:36.636879 25598 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb0de2dd430e46078a8686e349202b26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0de2dd430e46078a8686e349202b26" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41615 } }
I20260810 07:25:36.637259 25598 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb0de2dd430e46078a8686e349202b26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:36.637328 25598 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb0de2dd430e46078a8686e349202b26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb0de2dd430e46078a8686e349202b26, State: Initialized, Role: FOLLOWER
I20260810 07:25:36.638147 25598 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb0de2dd430e46078a8686e349202b26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0de2dd430e46078a8686e349202b26" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41615 } }
I20260810 07:25:36.638377 25598 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb0de2dd430e46078a8686e349202b26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:25:36.638465 25598 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb0de2dd430e46078a8686e349202b26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:25:36.638571 25598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb0de2dd430e46078a8686e349202b26 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:36.639459 25601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:36.624333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37342 (local address 127.12.157.254:41615)
0810 07:25:36.624804 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:36.624819 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:36.624857 (+ 38us) server_negotiation.cc:407] Connection header received
0810 07:25:36.625660 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:36.625679 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:36.626018 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:36.626356 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:36.627209 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:36.628314 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:36.629269 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:36.629835 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:36.630058 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:36.633433 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:36.633438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:36.633451 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:36.633479 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:36.635788 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:36.636367 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:36.636373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:36.636380 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:36.636673 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:36.637045 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:36.637049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:36.637050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:36.637628 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:36.637873 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:36.638154 (+ 281us) server_negotiation.cc:299] Negotiation successful
0810 07:25:36.638504 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":142,"threads_started":1}
I20260810 07:25:36.640239 25598 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb0de2dd430e46078a8686e349202b26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0de2dd430e46078a8686e349202b26" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41615 } }
I20260810 07:25:36.640758 25598 leader_election.cc:302] T 00000000000000000000000000000000 P eb0de2dd430e46078a8686e349202b26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb0de2dd430e46078a8686e349202b26; no voters:
I20260810 07:25:36.641466 25598 leader_election.cc:288] T 00000000000000000000000000000000 P eb0de2dd430e46078a8686e349202b26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:25:36.641546 25604 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb0de2dd430e46078a8686e349202b26 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:36.641775 25604 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb0de2dd430e46078a8686e349202b26 [term 1 LEADER]: Becoming Leader. State: Replica: eb0de2dd430e46078a8686e349202b26, State: Running, Role: LEADER
I20260810 07:25:36.642108 25604 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb0de2dd430e46078a8686e349202b26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0de2dd430e46078a8686e349202b26" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41615 } }
I20260810 07:25:36.643404 25604 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb0de2dd430e46078a8686e349202b26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb0de2dd430e46078a8686e349202b26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0de2dd430e46078a8686e349202b26" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41615 } } }
I20260810 07:25:36.643532 25604 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb0de2dd430e46078a8686e349202b26 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:36.643863 25598 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb0de2dd430e46078a8686e349202b26 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:25:36.643926 25605 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb0de2dd430e46078a8686e349202b26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb0de2dd430e46078a8686e349202b26. Latest consensus state: current_term: 1 leader_uuid: "eb0de2dd430e46078a8686e349202b26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0de2dd430e46078a8686e349202b26" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41615 } } }
I20260810 07:25:36.644024 25605 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb0de2dd430e46078a8686e349202b26 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:36.644320 25609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:25:36.647822 25609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:25:36.659605 25609 catalog_manager.cc:1347] Generated new cluster ID: 853fcf2ab7cf476da95fb31e81f4730b
I20260810 07:25:36.659698 25609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:25:36.679402 25609 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:25:36.681107 25609 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:25:36.695011 25609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb0de2dd430e46078a8686e349202b26: Generated new TSK 0
I20260810 07:25:36.695744 25609 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.193.keytab.
Entry for principal oryx/127.12.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.193.keytab.
I20260810 07:25:36.728421 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:0
--local_ip_for_outbound_sockets=127.12.157.193
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:41615
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.193.keytab
--principal=oryx/127.12.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:36.842173 25639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:36.842499 25639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:36.842576 25639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:36.847069 25639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:36.847159 25639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:36.847231 25639 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 07:25:36.847321 25639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:25:36.852869 25639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.193.keytab
--principal=oryx/127.12.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.157.254:41615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:36.854027 25639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:36.855152 25639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:36.865155 25652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:36.865155 25651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:36.865162 25661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:36.865290 25639 server_base.cc:1034] running on GCE node
I20260810 07:25:36.865852 25639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:36.866498 25639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:36.867697 25639 hybrid_clock.cc:648] HybridClock initialized: now 1786346736867664 us; error 45 us; skew 500 ppm
Aug 10 07:25:36 dist-test-slave-ffrd krb5kdc[25443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346736, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:36.873798 25639 init.cc:377] Logged in from keytab as oryx/127.12.157.193@KRBTEST.COM (short username oryx)
I20260810 07:25:36.874682 25639 webserver.cc:460] Webserver started at http://127.12.157.193:37121/ using document root <none> and password file <none>
I20260810 07:25:36.875262 25639 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:36.875370 25639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:36.875612 25639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:36.877810 25639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-0/data/instance:
uuid: "1f5b3dd52a30437bb9dc4217ecee6741"
format_stamp: "Formatted at 2026-08-10 07:25:36 on dist-test-slave-ffrd"
server_key: "181dadc7148896da93dcd6ce824fd8c3"
server_key_iv: "3c6a3b64b63180a89f00b9818c640071"
server_key_version: "encryptionkey@0"
I20260810 07:25:36.878400 25639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance:
uuid: "1f5b3dd52a30437bb9dc4217ecee6741"
format_stamp: "Formatted at 2026-08-10 07:25:36 on dist-test-slave-ffrd"
server_key: "181dadc7148896da93dcd6ce824fd8c3"
server_key_iv: "3c6a3b64b63180a89f00b9818c640071"
server_key_version: "encryptionkey@0"
I20260810 07:25:36.882145 25639 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260810 07:25:36.884734 25672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:36.886082 25639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:25:36.886253 25639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "1f5b3dd52a30437bb9dc4217ecee6741"
format_stamp: "Formatted at 2026-08-10 07:25:36 on dist-test-slave-ffrd"
server_key: "181dadc7148896da93dcd6ce824fd8c3"
server_key_iv: "3c6a3b64b63180a89f00b9818c640071"
server_key_version: "encryptionkey@0"
I20260810 07:25:36.886392 25639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:36.898633 25639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:36.902068 25639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:36.902310 25639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:36.903878 25639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:36.903968 25639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:36.904038 25639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:36.904086 25639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 07:25:36 dist-test-slave-ffrd krb5kdc[25443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346736, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM
I20260810 07:25:36.917048 25639 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:42927
I20260810 07:25:36.917181 25837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:42927 every 8 connection(s)
I20260810 07:25:36.918202 25639 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:25:36.920272 25601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:36.905052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:45829 (local address 127.12.157.254:41615)
0810 07:25:36.905201 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:36.905208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:36.906293 (+ 1085us) server_negotiation.cc:407] Connection header received
0810 07:25:36.907456 (+ 1163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:36.907463 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:36.907533 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:36.907630 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:36.909155 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:36.910074 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:36.911624 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:36.911923 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:36.912053 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:36.915694 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:36.915698 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:36.915701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:36.915737 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:36.917654 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:36.918381 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:36.918385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:36.918387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:36.918484 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:36.918998 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:36.919004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:36.919008 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:36.919284 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:36.919411 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:36.919952 (+ 541us) server_negotiation.cc:299] Negotiation successful
0810 07:25:36.920094 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 07:25:36.921108 25685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:36.905578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:41615 (local address 127.12.157.193:45829)
0810 07:25:36.906111 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:36.906156 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:36.907184 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:36.907885 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:36.907894 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:36.908424 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:36.908927 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:36.908996 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:36.910224 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:36.910228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:36.911490 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:36.911500 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:36.912112 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:36.912117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:36.912315 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:36.913034 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:36.913057 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:36.915439 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:36.917783 (+ 2344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:36.917792 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:36.917808 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:36.918171 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:36.918601 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:36.918605 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:36.918611 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:36.918843 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:36.919411 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:36.919427 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:36.919685 (+ 258us) client_negotiation.cc:769] Sending connection context
0810 07:25:36.919880 (+ 195us) client_negotiation.cc:240] Negotiation successful
0810 07:25:36.920220 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":150,"threads_started":1}
I20260810 07:25:36.922569 25838 heartbeater.cc:347] Connected to a master server at 127.12.157.254:41615
I20260810 07:25:36.922763 25838 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:36.923290 25838 heartbeater.cc:511] Master 127.12.157.254:41615 requested a full tablet report, sending...
I20260810 07:25:36.925122 25520 ts_manager.cc:194] Registered new tserver with Master: 1f5b3dd52a30437bb9dc4217ecee6741 (127.12.157.193:42927)
I20260810 07:25:36.925730 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 25639
I20260810 07:25:36.925879 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance
I20260810 07:25:36.926226 12919 external_mini_cluster.cc:1428] Setting key 323787ed3ea2bcf0b9f6fce4a865f2e9
I20260810 07:25:36.926739 25520 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.157.193@KRBTEST.COM'} at 127.12.157.193:45829
I20260810 07:25:36.933048 25850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:36.924757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:41615 (local address 127.12.157.193:51701)
0810 07:25:36.925153 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:36.925170 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:36.925285 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:36.925575 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:36.925577 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:36.925804 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:36.925947 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:36.925955 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:36.927001 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:36.927005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:36.927808 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:36.927818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:36.928299 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:36.928303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:36.928444 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:36.929010 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:36.929023 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:36.929366 (+ 343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:36.931469 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:36.931472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:36.931474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:36.931676 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:36.931984 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:36.931987 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:36.931988 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:36.932066 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:36.932529 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:36.932531 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:36.932608 (+ 77us) client_negotiation.cc:769] Sending connection context
0810 07:25:36.932688 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 07:25:36.932810 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":1,"thread_start_us":119,"threads_started":1}
I20260810 07:25:36.933357 25601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:36.924849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:51701 (local address 127.12.157.254:41615)
0810 07:25:36.925011 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:36.925018 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:36.925260 (+ 242us) server_negotiation.cc:407] Connection header received
0810 07:25:36.925371 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:36.925374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:36.925414 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:36.925537 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:36.926089 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:36.926806 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:36.927942 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:36.928178 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:36.928295 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:36.929517 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:36.929521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:36.929523 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:36.929548 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:36.931316 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:36.931780 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:36.931787 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:36.931791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:36.931887 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:36.932160 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:36.932166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:36.932171 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:36.932430 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:36.932539 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:36.932787 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 07:25:36.932925 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":61}
WARNING: no policy specified for oryx/127.12.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.194.keytab.
Entry for principal oryx/127.12.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.194.keytab.
I20260810 07:25:36.958129 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:0
--local_ip_for_outbound_sockets=127.12.157.194
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:41615
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.194.keytab
--principal=oryx/127.12.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:37.076045 25855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:37.076359 25855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:37.076436 25855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:37.081076 25855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:37.081156 25855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:37.081220 25855 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 07:25:37.081305 25855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:25:37.087014 25855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.194.keytab
--principal=oryx/127.12.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.157.254:41615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:37.088088 25855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:37.089218 25855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:37.098932 25865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:37.098960 25867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:37.098939 25864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:37.099393 25855 server_base.cc:1034] running on GCE node
I20260810 07:25:37.099821 25855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:37.100449 25855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:37.101617 25855 hybrid_clock.cc:648] HybridClock initialized: now 1786346737101582 us; error 47 us; skew 500 ppm
Aug 10 07:25:37 dist-test-slave-ffrd krb5kdc[25443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346737, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:37.104372 25855 init.cc:377] Logged in from keytab as oryx/127.12.157.194@KRBTEST.COM (short username oryx)
I20260810 07:25:37.105378 25855 webserver.cc:460] Webserver started at http://127.12.157.194:46023/ using document root <none> and password file <none>
I20260810 07:25:37.105944 25855 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:37.106041 25855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:37.106256 25855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:37.108366 25855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-1/data/instance:
uuid: "03e839a9c2594f89b0e4f6cddff8a193"
format_stamp: "Formatted at 2026-08-10 07:25:37 on dist-test-slave-ffrd"
server_key: "0571bc5717da53d2d93214133ddd8114"
server_key_iv: "e7dd8d697e3f4589961ee21834d7d250"
server_key_version: "encryptionkey@0"
I20260810 07:25:37.108983 25855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance:
uuid: "03e839a9c2594f89b0e4f6cddff8a193"
format_stamp: "Formatted at 2026-08-10 07:25:37 on dist-test-slave-ffrd"
server_key: "0571bc5717da53d2d93214133ddd8114"
server_key_iv: "e7dd8d697e3f4589961ee21834d7d250"
server_key_version: "encryptionkey@0"
I20260810 07:25:37.112572 25855 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:25:37.115036 25875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:37.116219 25855 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 07:25:37.116367 25855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "03e839a9c2594f89b0e4f6cddff8a193"
format_stamp: "Formatted at 2026-08-10 07:25:37 on dist-test-slave-ffrd"
server_key: "0571bc5717da53d2d93214133ddd8114"
server_key_iv: "e7dd8d697e3f4589961ee21834d7d250"
server_key_version: "encryptionkey@0"
I20260810 07:25:37.116480 25855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:37.136581 25855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:37.139873 25855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:37.140089 25855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:37.141717 25855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:37.141798 25855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:37.141867 25855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:37.141901 25855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 07:25:37 dist-test-slave-ffrd krb5kdc[25443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346737, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM
I20260810 07:25:37.153810 25855 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:38095
I20260810 07:25:37.153915 26057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:38095 every 8 connection(s)
I20260810 07:25:37.154750 25855 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:25:37.155658 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 25855
I20260810 07:25:37.155802 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance
I20260810 07:25:37.156069 12919 external_mini_cluster.cc:1428] Setting key 2f5b967d3df079f8f3183e3917f7ab3e
I20260810 07:25:37.157372 25601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:37.142474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:48945 (local address 127.12.157.254:41615)
0810 07:25:37.142631 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:37.142636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:37.143654 (+ 1018us) server_negotiation.cc:407] Connection header received
0810 07:25:37.145025 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:37.145032 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:37.145102 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:37.145207 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:37.146667 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.147196 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.148395 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.148731 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.148809 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:37.153113 (+ 4304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:37.153117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:37.153119 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:37.153147 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:37.154871 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:37.155493 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:37.155498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:37.155500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:37.155593 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:37.156023 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:37.156028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:37.156029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:37.156234 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:37.156351 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:37.157079 (+ 728us) server_negotiation.cc:299] Negotiation successful
0810 07:25:37.157214 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 07:25:37.158102 25890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:37.142937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:41615 (local address 127.12.157.194:48945)
0810 07:25:37.143459 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:37.143521 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:37.144764 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:37.145445 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:37.145454 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:37.145822 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:37.146509 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:37.146531 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.147332 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.147336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:37.148225 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:37.148235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.149061 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.149064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:37.149208 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:37.149869 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:37.149897 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:37.152911 (+ 3014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:37.155001 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:37.155007 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:37.155021 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:37.155365 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:37.155696 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:37.155700 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:37.155702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:37.155915 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:37.156351 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:37.156359 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:37.156668 (+ 309us) client_negotiation.cc:769] Sending connection context
0810 07:25:37.156872 (+ 204us) client_negotiation.cc:240] Negotiation successful
0810 07:25:37.157240 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":115,"threads_started":1}
I20260810 07:25:37.159385 26058 heartbeater.cc:347] Connected to a master server at 127.12.157.254:41615
I20260810 07:25:37.159523 26058 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:37.160074 26058 heartbeater.cc:511] Master 127.12.157.254:41615 requested a full tablet report, sending...
I20260810 07:25:37.161213 25520 ts_manager.cc:194] Registered new tserver with Master: 03e839a9c2594f89b0e4f6cddff8a193 (127.12.157.194:38095)
I20260810 07:25:37.162024 25520 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.157.194@KRBTEST.COM'} at 127.12.157.194:48945
WARNING: no policy specified for oryx/127.12.157.195@KRBTEST.COM; defaulting to no policy
I20260810 07:25:37.168928 26070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:37.161745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:41615 (local address 127.12.157.194:56461)
0810 07:25:37.162064 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:37.162075 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:37.162231 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:37.162533 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:37.162536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:37.162745 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:37.162856 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:37.162861 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.163619 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.163622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:37.164182 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:37.164190 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.164662 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.164666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:37.164756 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:37.165337 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:37.165349 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:37.165658 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:37.167356 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:37.167361 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:37.167363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:37.167561 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:37.167899 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:37.167904 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:37.167908 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:37.168015 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:37.168433 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:37.168435 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:37.168513 (+ 78us) client_negotiation.cc:769] Sending connection context
0810 07:25:37.168611 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 07:25:37.168719 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":99,"threads_started":1}
I20260810 07:25:37.169140 25601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:37.161787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:56461 (local address 127.12.157.254:41615)
0810 07:25:37.161905 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:37.161910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:37.162162 (+ 252us) server_negotiation.cc:407] Connection header received
0810 07:25:37.162355 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:37.162360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:37.162413 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:37.162499 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:37.162975 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.163490 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.164306 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.164541 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.164657 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:37.165801 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:37.165805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:37.165807 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:37.165831 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:37.167233 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:37.167666 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:37.167669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:37.167671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:37.167755 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:37.168122 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:37.168125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:37.168126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:37.168334 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:37.168414 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:37.168706 (+ 292us) server_negotiation.cc:299] Negotiation successful
0810 07:25:37.168941 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.195.keytab.
Entry for principal oryx/127.12.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.195.keytab.
I20260810 07:25:37.188011 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:0
--local_ip_for_outbound_sockets=127.12.157.195
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:41615
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.195.keytab
--principal=oryx/127.12.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:36847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:37.303224 26072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:37.303481 26072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:37.303558 26072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:37.307595 26072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:37.307660 26072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:37.307699 26072 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 07:25:37.307799 26072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:25:37.313318 26072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:36847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.195.keytab
--principal=oryx/127.12.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.157.254:41615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:37.314460 26072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:37.315527 26072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:37.325685 26085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:37.325688 26083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:37.325688 26087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:37.326081 26072 server_base.cc:1034] running on GCE node
I20260810 07:25:37.326465 26072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:37.327108 26072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:37.328336 26072 hybrid_clock.cc:648] HybridClock initialized: now 1786346737328320 us; error 34 us; skew 500 ppm
Aug 10 07:25:37 dist-test-slave-ffrd krb5kdc[25443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346737, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:37.331540 26072 init.cc:377] Logged in from keytab as oryx/127.12.157.195@KRBTEST.COM (short username oryx)
I20260810 07:25:37.332551 26072 webserver.cc:460] Webserver started at http://127.12.157.195:42125/ using document root <none> and password file <none>
I20260810 07:25:37.333266 26072 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:37.333374 26072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:37.333695 26072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:37.336177 26072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-2/data/instance:
uuid: "96142be645164fd7887695f0986f8965"
format_stamp: "Formatted at 2026-08-10 07:25:37 on dist-test-slave-ffrd"
server_key: "4e18a6c1fd4d81f3d4a1492fa6da5f34"
server_key_iv: "8829f9d3db122f944d351a741e107071"
server_key_version: "encryptionkey@0"
I20260810 07:25:37.336824 26072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance:
uuid: "96142be645164fd7887695f0986f8965"
format_stamp: "Formatted at 2026-08-10 07:25:37 on dist-test-slave-ffrd"
server_key: "4e18a6c1fd4d81f3d4a1492fa6da5f34"
server_key_iv: "8829f9d3db122f944d351a741e107071"
server_key_version: "encryptionkey@0"
I20260810 07:25:37.340881 26072 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260810 07:25:37.343590 26098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:37.345124 26072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 07:25:37.345245 26072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "96142be645164fd7887695f0986f8965"
format_stamp: "Formatted at 2026-08-10 07:25:37 on dist-test-slave-ffrd"
server_key: "4e18a6c1fd4d81f3d4a1492fa6da5f34"
server_key_iv: "8829f9d3db122f944d351a741e107071"
server_key_version: "encryptionkey@0"
I20260810 07:25:37.345361 26072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:37.374042 26072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:37.377897 26072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:37.378120 26072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:37.379724 26072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:37.379779 26072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:37.379823 26072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:37.379878 26072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 07:25:37 dist-test-slave-ffrd krb5kdc[25443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346737, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM
I20260810 07:25:37.391521 26072 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:41691
I20260810 07:25:37.391810 26279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:41691 every 8 connection(s)
I20260810 07:25:37.392544 26072 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:25:37.394809 25601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:37.380483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:47545 (local address 127.12.157.254:41615)
0810 07:25:37.380642 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:37.380647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:37.381708 (+ 1061us) server_negotiation.cc:407] Connection header received
0810 07:25:37.383166 (+ 1458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:37.383170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:37.383241 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:37.383351 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:37.384617 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.385224 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.386477 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.386698 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.386819 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:37.390516 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:37.390519 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:37.390522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:37.390553 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:37.392337 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:37.392982 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:37.392989 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:37.392993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:37.393122 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:37.393721 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:37.393724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:37.393725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:37.393926 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:37.394017 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:37.394559 (+ 542us) server_negotiation.cc:299] Negotiation successful
0810 07:25:37.394679 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 07:25:37.395618 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:37.380987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:41615 (local address 127.12.157.195:47545)
0810 07:25:37.381517 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:37.381579 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:37.382808 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:37.383592 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:37.383601 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:37.383975 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:37.384460 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:37.384474 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.385357 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.385360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:37.386318 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:37.386329 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.386876 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.386881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:37.387104 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:37.387857 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:37.387888 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:37.390335 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:37.392481 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:37.392487 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:37.392501 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:37.392851 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:37.393452 (+ 601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:37.393456 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:37.393458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:37.393611 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:37.394055 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:37.394060 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:37.394305 (+ 245us) client_negotiation.cc:769] Sending connection context
0810 07:25:37.394491 (+ 186us) client_negotiation.cc:240] Negotiation successful
0810 07:25:37.394746 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":115,"threads_started":1}
I20260810 07:25:37.396155 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 26072
I20260810 07:25:37.396303 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance
I20260810 07:25:37.396634 12919 external_mini_cluster.cc:1428] Setting key 64328cebd767abd9fe8b63058cf0751e
I20260810 07:25:37.397073 26280 heartbeater.cc:347] Connected to a master server at 127.12.157.254:41615
I20260810 07:25:37.397248 26280 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:37.397882 26280 heartbeater.cc:511] Master 127.12.157.254:41615 requested a full tablet report, sending...
I20260810 07:25:37.399008 25522 ts_manager.cc:194] Registered new tserver with Master: 96142be645164fd7887695f0986f8965 (127.12.157.195:41691)
I20260810 07:25:37.399801 25522 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.157.195@KRBTEST.COM'} at 127.12.157.195:47545
I20260810 07:25:37.406668 25601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:37.399464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:39145 (local address 127.12.157.254:41615)
0810 07:25:37.399582 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:37.399589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:37.399789 (+ 200us) server_negotiation.cc:407] Connection header received
0810 07:25:37.399909 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:37.399912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:37.399955 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:37.400047 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:37.400499 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.401097 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.401932 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.402166 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.402281 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:37.403433 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:37.403436 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:37.403439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:37.403463 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:37.404925 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:37.405412 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:37.405417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:37.405418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:37.405503 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:37.405821 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:37.405825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:37.405827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:37.406013 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:37.406127 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:37.406402 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 07:25:37.406512 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 07:25:37.406680 26290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:37.399426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:41615 (local address 127.12.157.195:39145)
0810 07:25:37.399705 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:37.399717 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:37.399830 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:37.400063 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:37.400066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:37.400276 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:37.400380 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:37.400386 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.401227 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.401230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:37.401801 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:37.401809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.402286 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.402289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:37.402397 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:37.402927 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:37.402939 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:37.403270 (+ 331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:37.405091 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:37.405095 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:37.405098 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:37.405301 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:37.405626 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:37.405630 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:37.405631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:37.405722 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:37.406117 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:37.406121 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:37.406220 (+ 99us) client_negotiation.cc:769] Sending connection context
0810 07:25:37.406331 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 07:25:37.406495 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":113,"threads_started":1}
I20260810 07:25:37.410697 12919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 07:25:37 dist-test-slave-ffrd krb5kdc[25443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346737, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 07:25:37 dist-test-slave-ffrd krb5kdc[25443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:37 dist-test-slave-ffrd krb5kdc[25443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 07:25:37.438731 25601 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:37.431648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39284 (local address 127.12.157.254:41615)
0810 07:25:37.431792 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:37.431796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:37.431808 (+ 12us) server_negotiation.cc:407] Connection header received
0810 07:25:37.431852 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:37.431855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:37.431895 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:37.432001 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:37.432810 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.433490 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.434501 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.434756 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.434910 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:37.438573 (+ 3663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39284: BlockingRecv error: recv got EOF from 127.0.0.1:39284 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 07:25:37 dist-test-slave-ffrd krb5kdc[25443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346737, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM
I20260810 07:25:37.449329 25601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:37.440380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39288 (local address 127.12.157.254:41615)
0810 07:25:37.440564 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:37.440567 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:37.440641 (+ 74us) server_negotiation.cc:407] Connection header received
0810 07:25:37.440734 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:37.440736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:37.440778 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:37.440849 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:37.441582 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.442125 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.442880 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.443104 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.443186 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:37.445703 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:37.445706 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:37.445708 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:37.445732 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:37.447313 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:37.447789 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:37.447793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:37.447794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:37.447923 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:37.448250 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:37.448252 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:37.448253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:37.448438 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:37.448623 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:37.448907 (+ 284us) server_negotiation.cc:299] Negotiation successful
0810 07:25:37.449060 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260810 07:25:37.452396 25522 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:25:37.455554 25522 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:25:37.470675 26317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:37.464693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:41691 (local address 127.0.0.1:58434)
0810 07:25:37.465692 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:37.465703 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:37.465771 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:37.466463 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:37.466472 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:37.466500 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:37.466663 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:37.466670 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.467709 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.467715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:37.469133 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:37.469141 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.470046 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.470050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:37.470236 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:37.470285 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 07:25:37.470394 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 07:25:37.470513 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"mutex_wait_us":46,"spinlock_wait_cycles":1280,"thread_start_us":94,"threads_started":1}
I20260810 07:25:37.470709 26322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:37.465254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58434 (local address 127.12.157.195:41691)
0810 07:25:37.465910 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:37.465922 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:37.465954 (+ 32us) server_negotiation.cc:407] Connection header received
0810 07:25:37.466019 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:37.466023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:37.466262 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:37.466477 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:37.466813 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.467587 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.469264 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.469913 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.470083 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:37.470237 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:37.470437 (+ 200us) server_negotiation.cc:299] Negotiation successful
0810 07:25:37.470546 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":114,"threads_started":1}
I20260810 07:25:37.471063 26315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:37.464449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:38095 (local address 127.0.0.1:38940)
0810 07:25:37.465236 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:37.465299 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:37.465441 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:37.467171 (+ 1730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:37.467174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:37.467192 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:37.467274 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:37.467280 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.468225 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.468231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:37.469367 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:37.469378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.470662 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.470666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:37.470764 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:37.470791 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 07:25:37.470867 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 07:25:37.470932 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":78,"threads_started":1}
I20260810 07:25:37.471550 26324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:37.465203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38940 (local address 127.12.157.194:38095)
0810 07:25:37.466806 (+ 1603us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:37.466819 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:37.466838 (+ 19us) server_negotiation.cc:407] Connection header received
0810 07:25:37.466912 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:37.466916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:37.467040 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:37.467195 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:37.467371 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.468100 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.469507 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.470323 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.470472 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:37.470595 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:37.471294 (+ 699us) server_negotiation.cc:299] Negotiation successful
0810 07:25:37.471412 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1488,"thread_start_us":180,"threads_started":1}
I20260810 07:25:37.472142 26316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:37.464486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:42927 (local address 127.0.0.1:41500)
0810 07:25:37.465254 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:37.465304 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:37.465441 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:37.466524 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:37.466527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:37.466554 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:37.466667 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:37.466673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.469088 (+ 2415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.469091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:37.470236 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:37.470249 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.471483 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.471486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:37.471550 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:37.471567 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 07:25:37.471937 (+ 370us) client_negotiation.cc:240] Negotiation successful
0810 07:25:37.472014 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":37,"spinlock_wait_cycles":768,"thread_start_us":92,"threads_started":1}
I20260810 07:25:37.472656 26321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:37.464728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41500 (local address 127.12.157.193:42927)
0810 07:25:37.466023 (+ 1295us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:37.466030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:37.466055 (+ 25us) server_negotiation.cc:407] Connection header received
0810 07:25:37.466141 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:37.466145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:37.466359 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:37.466530 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:37.467759 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.468910 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.470355 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.471181 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.471311 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:37.471428 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:37.472418 (+ 990us) server_negotiation.cc:299] Negotiation successful
0810 07:25:37.472526 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1186,"thread_start_us":98,"threads_started":1}
I20260810 07:25:37.474237 25957 tablet_service.cc:1467] Processing CreateTablet for tablet 1076772bd258417e8199ca18605a24f2 (DEFAULT_TABLE table=test-table [id=326ad8e6ebf84d2da1779d0034c67ddf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:37.474236 26177 tablet_service.cc:1467] Processing CreateTablet for tablet 1076772bd258417e8199ca18605a24f2 (DEFAULT_TABLE table=test-table [id=326ad8e6ebf84d2da1779d0034c67ddf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:37.475430 26177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1076772bd258417e8199ca18605a24f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:37.475421 25957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1076772bd258417e8199ca18605a24f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:37.477435 25742 tablet_service.cc:1467] Processing CreateTablet for tablet 1076772bd258417e8199ca18605a24f2 (DEFAULT_TABLE table=test-table [id=326ad8e6ebf84d2da1779d0034c67ddf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:37.478375 25742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1076772bd258417e8199ca18605a24f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:37.485059 26331 tablet_bootstrap.cc:492] T 1076772bd258417e8199ca18605a24f2 P 1f5b3dd52a30437bb9dc4217ecee6741: Bootstrap starting.
I20260810 07:25:37.487104 26328 tablet_bootstrap.cc:492] T 1076772bd258417e8199ca18605a24f2 P 03e839a9c2594f89b0e4f6cddff8a193: Bootstrap starting.
I20260810 07:25:37.487469 26331 tablet_bootstrap.cc:654] T 1076772bd258417e8199ca18605a24f2 P 1f5b3dd52a30437bb9dc4217ecee6741: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:37.487553 26325 tablet_bootstrap.cc:492] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965: Bootstrap starting.
I20260810 07:25:37.488444 26331 log.cc:824] T 1076772bd258417e8199ca18605a24f2 P 1f5b3dd52a30437bb9dc4217ecee6741: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:37.489848 26325 tablet_bootstrap.cc:654] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:37.490713 26325 log.cc:824] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:37.491348 26328 tablet_bootstrap.cc:654] T 1076772bd258417e8199ca18605a24f2 P 03e839a9c2594f89b0e4f6cddff8a193: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:37.492184 26328 log.cc:824] T 1076772bd258417e8199ca18605a24f2 P 03e839a9c2594f89b0e4f6cddff8a193: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:37.492643 26331 tablet_bootstrap.cc:492] T 1076772bd258417e8199ca18605a24f2 P 1f5b3dd52a30437bb9dc4217ecee6741: No bootstrap required, opened a new log
I20260810 07:25:37.492882 26331 ts_tablet_manager.cc:1397] T 1076772bd258417e8199ca18605a24f2 P 1f5b3dd52a30437bb9dc4217ecee6741: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260810 07:25:37.493772 26325 tablet_bootstrap.cc:492] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965: No bootstrap required, opened a new log
I20260810 07:25:37.493988 26325 ts_tablet_manager.cc:1397] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 07:25:37.496976 26331 raft_consensus.cc:348] T 1076772bd258417e8199ca18605a24f2 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:37.497032 26325 raft_consensus.cc:348] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:37.497242 26331 raft_consensus.cc:374] T 1076772bd258417e8199ca18605a24f2 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:37.497311 26331 raft_consensus.cc:729] T 1076772bd258417e8199ca18605a24f2 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f5b3dd52a30437bb9dc4217ecee6741, State: Initialized, Role: FOLLOWER
I20260810 07:25:37.497298 26325 raft_consensus.cc:374] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:37.497440 26325 raft_consensus.cc:729] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96142be645164fd7887695f0986f8965, State: Initialized, Role: FOLLOWER
I20260810 07:25:37.497831 26331 consensus_queue.cc:260] T 1076772bd258417e8199ca18605a24f2 P 1f5b3dd52a30437bb9dc4217ecee6741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:37.497908 26325 consensus_queue.cc:260] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:37.498507 25838 heartbeater.cc:503] Master 127.12.157.254:41615 was elected leader, sending a full tablet report...
I20260810 07:25:37.498838 26325 ts_tablet_manager.cc:1428] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 07:25:37.498847 26331 ts_tablet_manager.cc:1428] T 1076772bd258417e8199ca18605a24f2 P 1f5b3dd52a30437bb9dc4217ecee6741: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 07:25:37.499416 26328 tablet_bootstrap.cc:492] T 1076772bd258417e8199ca18605a24f2 P 03e839a9c2594f89b0e4f6cddff8a193: No bootstrap required, opened a new log
I20260810 07:25:37.499629 26328 ts_tablet_manager.cc:1397] T 1076772bd258417e8199ca18605a24f2 P 03e839a9c2594f89b0e4f6cddff8a193: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260810 07:25:37.499622 26280 heartbeater.cc:503] Master 127.12.157.254:41615 was elected leader, sending a full tablet report...
I20260810 07:25:37.503365 26328 raft_consensus.cc:348] T 1076772bd258417e8199ca18605a24f2 P 03e839a9c2594f89b0e4f6cddff8a193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:37.503594 26328 raft_consensus.cc:374] T 1076772bd258417e8199ca18605a24f2 P 03e839a9c2594f89b0e4f6cddff8a193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:37.503628 26328 raft_consensus.cc:729] T 1076772bd258417e8199ca18605a24f2 P 03e839a9c2594f89b0e4f6cddff8a193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e839a9c2594f89b0e4f6cddff8a193, State: Initialized, Role: FOLLOWER
I20260810 07:25:37.504112 26328 consensus_queue.cc:260] T 1076772bd258417e8199ca18605a24f2 P 03e839a9c2594f89b0e4f6cddff8a193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:37.504891 26058 heartbeater.cc:503] Master 127.12.157.254:41615 was elected leader, sending a full tablet report...
I20260810 07:25:37.505231 26328 ts_tablet_manager.cc:1428] T 1076772bd258417e8199ca18605a24f2 P 03e839a9c2594f89b0e4f6cddff8a193: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 07:25:37.553450 26341 raft_consensus.cc:482] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:25:37.553659 26341 raft_consensus.cc:504] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:37.554817 26341 leader_election.cc:288] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03e839a9c2594f89b0e4f6cddff8a193 (127.12.157.194:38095), 1f5b3dd52a30437bb9dc4217ecee6741 (127.12.157.193:42927)
I20260810 07:25:37.559379 26343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:37.555099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:38095 (local address 127.12.157.195:42997)
0810 07:25:37.555769 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:37.555783 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:37.555891 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:37.556147 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:37.556150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:37.556168 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:37.556296 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:37.556303 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.557199 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.557202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:37.558023 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:37.558032 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.558876 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.558898 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:37.559013 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:37.559042 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 07:25:37.559122 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 07:25:37.559193 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":256,"thread_start_us":166,"threads_started":1}
I20260810 07:25:37.559424 26324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:37.555293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:42997 (local address 127.12.157.194:38095)
0810 07:25:37.555488 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:37.555492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:37.555902 (+ 410us) server_negotiation.cc:407] Connection header received
0810 07:25:37.555967 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:37.555973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:37.556033 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:37.556103 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:37.556421 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.557077 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.558153 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.558741 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.558885 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:37.558928 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:37.559185 (+ 257us) server_negotiation.cc:299] Negotiation successful
0810 07:25:37.559254 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 07:25:37.559921 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:37.555099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:42927 (local address 127.12.157.195:46537)
0810 07:25:37.555273 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:37.555289 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:37.555402 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:37.555742 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:37.555745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:37.555764 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:37.555920 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:37.555930 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.556835 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.556839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:37.558512 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:37.558524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.559440 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.559447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:37.559576 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:37.559606 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 07:25:37.559695 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 07:25:37.559769 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 07:25:37.559970 26321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:37.555186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:46537 (local address 127.12.157.193:42927)
0810 07:25:37.555361 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:37.555368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:37.555398 (+ 30us) server_negotiation.cc:407] Connection header received
0810 07:25:37.555543 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:37.555546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:37.555624 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:37.555716 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:37.556048 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.556715 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.558652 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.559320 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.559456 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:37.559513 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:37.559746 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 07:25:37.559821 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 07:25:37.560101 25982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1076772bd258417e8199ca18605a24f2" candidate_uuid: "96142be645164fd7887695f0986f8965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e839a9c2594f89b0e4f6cddff8a193" is_pre_election: true
I20260810 07:25:37.560490 25982 raft_consensus.cc:2454] T 1076772bd258417e8199ca18605a24f2 P 03e839a9c2594f89b0e4f6cddff8a193 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96142be645164fd7887695f0986f8965 in term 0.
I20260810 07:25:37.560925 25766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1076772bd258417e8199ca18605a24f2" candidate_uuid: "96142be645164fd7887695f0986f8965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" is_pre_election: true
I20260810 07:25:37.561064 26099 leader_election.cc:302] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e839a9c2594f89b0e4f6cddff8a193, 96142be645164fd7887695f0986f8965; no voters:
I20260810 07:25:37.561239 25766 raft_consensus.cc:2454] T 1076772bd258417e8199ca18605a24f2 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96142be645164fd7887695f0986f8965 in term 0.
I20260810 07:25:37.561371 26341 raft_consensus.cc:2781] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:25:37.561470 26341 raft_consensus.cc:482] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:25:37.561517 26341 raft_consensus.cc:3037] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:37.562772 26341 raft_consensus.cc:504] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:37.563135 26341 leader_election.cc:288] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965 [CANDIDATE]: Term 1 election: Requested vote from peers 03e839a9c2594f89b0e4f6cddff8a193 (127.12.157.194:38095), 1f5b3dd52a30437bb9dc4217ecee6741 (127.12.157.193:42927)
I20260810 07:25:37.563541 25982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1076772bd258417e8199ca18605a24f2" candidate_uuid: "96142be645164fd7887695f0986f8965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e839a9c2594f89b0e4f6cddff8a193"
I20260810 07:25:37.563556 25766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1076772bd258417e8199ca18605a24f2" candidate_uuid: "96142be645164fd7887695f0986f8965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f5b3dd52a30437bb9dc4217ecee6741"
I20260810 07:25:37.563653 25766 raft_consensus.cc:3037] T 1076772bd258417e8199ca18605a24f2 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:37.563685 25982 raft_consensus.cc:3037] T 1076772bd258417e8199ca18605a24f2 P 03e839a9c2594f89b0e4f6cddff8a193 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:37.564747 25982 raft_consensus.cc:2454] T 1076772bd258417e8199ca18605a24f2 P 03e839a9c2594f89b0e4f6cddff8a193 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96142be645164fd7887695f0986f8965 in term 1.
I20260810 07:25:37.564749 25766 raft_consensus.cc:2454] T 1076772bd258417e8199ca18605a24f2 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96142be645164fd7887695f0986f8965 in term 1.
I20260810 07:25:37.565105 26100 leader_election.cc:302] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f5b3dd52a30437bb9dc4217ecee6741, 96142be645164fd7887695f0986f8965; no voters:
I20260810 07:25:37.565311 26341 raft_consensus.cc:2781] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:37.565560 26341 raft_consensus.cc:686] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965 [term 1 LEADER]: Becoming Leader. State: Replica: 96142be645164fd7887695f0986f8965, State: Running, Role: LEADER
I20260810 07:25:37.565923 26341 consensus_queue.cc:237] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:37.569343 25520 catalog_manager.cc:5495] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96142be645164fd7887695f0986f8965 (127.12.157.195). New cstate: current_term: 1 leader_uuid: "96142be645164fd7887695f0986f8965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } health_report { overall_health: HEALTHY } } }
I20260810 07:25:37.595046 26322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:37.591340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58446 (local address 127.12.157.195:41691)
0810 07:25:37.591588 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:37.591592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:37.591607 (+ 15us) server_negotiation.cc:407] Connection header received
0810 07:25:37.591690 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:37.591692 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:37.591749 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:37.591849 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:25:37.592141 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.592678 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.593614 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.593831 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.593909 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:37.594191 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:25:37.594665 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:25:37.594752 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:37.594872 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 07:25:37.594926 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260810 07:25:37.601962 25766 raft_consensus.cc:1261] T 1076772bd258417e8199ca18605a24f2 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 1 FOLLOWER]: Refusing update from remote peer 96142be645164fd7887695f0986f8965: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:37.602394 25982 raft_consensus.cc:1261] T 1076772bd258417e8199ca18605a24f2 P 03e839a9c2594f89b0e4f6cddff8a193 [term 1 FOLLOWER]: Refusing update from remote peer 96142be645164fd7887695f0986f8965: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:37.602762 26344 consensus_queue.cc:1035] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:37.603053 26341 consensus_queue.cc:1035] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:37.613660 26348 mvcc.cc:205] Tried to move back new op lower bound from 7316876237209452544 to 7316876237072764928. Current Snapshot: MvccSnapshot[applied={T|T < 7316876237209452544}]
I20260810 07:25:37.620409 25519 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:37.620637 25519 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:37.623553 25519 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb0de2dd430e46078a8686e349202b26: Sending DeleteTablet for 3 replicas of tablet 1076772bd258417e8199ca18605a24f2
I20260810 07:25:37.624588 25742 tablet_service.cc:1514] Processing DeleteTablet for tablet 1076772bd258417e8199ca18605a24f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:37 UTC) from {username='oryx', principal='oryx/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:41500
I20260810 07:25:37.624600 26177 tablet_service.cc:1514] Processing DeleteTablet for tablet 1076772bd258417e8199ca18605a24f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:37 UTC) from {username='oryx', principal='oryx/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:58434
I20260810 07:25:37.625134 25957 tablet_service.cc:1514] Processing DeleteTablet for tablet 1076772bd258417e8199ca18605a24f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:37 UTC) from {username='oryx', principal='oryx/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:38940
I20260810 07:25:37.625626 26365 tablet_replica.cc:330] T 1076772bd258417e8199ca18605a24f2 P 03e839a9c2594f89b0e4f6cddff8a193: stopping tablet replica
I20260810 07:25:37.625990 26365 raft_consensus.cc:2229] T 1076772bd258417e8199ca18605a24f2 P 03e839a9c2594f89b0e4f6cddff8a193 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:25:37.626340 26365 raft_consensus.cc:2258] T 1076772bd258417e8199ca18605a24f2 P 03e839a9c2594f89b0e4f6cddff8a193 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:37.626605 25519 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:25:37.627031 25519 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:25:37.628098 26365 ts_tablet_manager.cc:1905] T 1076772bd258417e8199ca18605a24f2 P 03e839a9c2594f89b0e4f6cddff8a193: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:37.631372 26361 tablet_replica.cc:330] T 1076772bd258417e8199ca18605a24f2 P 1f5b3dd52a30437bb9dc4217ecee6741: stopping tablet replica
I20260810 07:25:37.631502 26365 ts_tablet_manager.cc:1918] T 1076772bd258417e8199ca18605a24f2 P 03e839a9c2594f89b0e4f6cddff8a193: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:25:37.631615 26365 log.cc:1196] T 1076772bd258417e8199ca18605a24f2 P 03e839a9c2594f89b0e4f6cddff8a193: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-1/wal/wals/1076772bd258417e8199ca18605a24f2
I20260810 07:25:37.631985 26365 ts_tablet_manager.cc:1939] T 1076772bd258417e8199ca18605a24f2 P 03e839a9c2594f89b0e4f6cddff8a193: Deleting consensus metadata
I20260810 07:25:37.632009 26361 raft_consensus.cc:2229] T 1076772bd258417e8199ca18605a24f2 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:25:37.632859 26361 raft_consensus.cc:2258] T 1076772bd258417e8199ca18605a24f2 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:37.633819 26362 tablet_replica.cc:330] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965: stopping tablet replica
I20260810 07:25:37.633855 25495 catalog_manager.cc:4841] TS 03e839a9c2594f89b0e4f6cddff8a193 (127.12.157.194:38095): tablet 1076772bd258417e8199ca18605a24f2 (table test-table [id=326ad8e6ebf84d2da1779d0034c67ddf]) successfully deleted
I20260810 07:25:37.634270 26362 raft_consensus.cc:2229] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:25:37.634395 26177 tablet_service.cc:1467] Processing CreateTablet for tablet c038ed35747d4cccb7e474f10026d679 (DEFAULT_TABLE table=test-table [id=d5dfbd6c2753401b9ee37d7257c704a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:37.634475 25742 tablet_service.cc:1467] Processing CreateTablet for tablet c038ed35747d4cccb7e474f10026d679 (DEFAULT_TABLE table=test-table [id=d5dfbd6c2753401b9ee37d7257c704a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:37.634685 26177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c038ed35747d4cccb7e474f10026d679. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:37.634773 25742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c038ed35747d4cccb7e474f10026d679. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:37.635038 26362 raft_consensus.cc:2258] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:37.636991 26331 tablet_bootstrap.cc:492] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741: Bootstrap starting.
I20260810 07:25:37.637279 25957 tablet_service.cc:1467] Processing CreateTablet for tablet c038ed35747d4cccb7e474f10026d679 (DEFAULT_TABLE table=test-table [id=d5dfbd6c2753401b9ee37d7257c704a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:37.637583 25957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c038ed35747d4cccb7e474f10026d679. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:37.638053 26331 tablet_bootstrap.cc:654] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:37.638775 26325 tablet_bootstrap.cc:492] T c038ed35747d4cccb7e474f10026d679 P 96142be645164fd7887695f0986f8965: Bootstrap starting.
I20260810 07:25:37.639883 26325 tablet_bootstrap.cc:654] T c038ed35747d4cccb7e474f10026d679 P 96142be645164fd7887695f0986f8965: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:37.640162 26328 tablet_bootstrap.cc:492] T c038ed35747d4cccb7e474f10026d679 P 03e839a9c2594f89b0e4f6cddff8a193: Bootstrap starting.
I20260810 07:25:37.641206 26328 tablet_bootstrap.cc:654] T c038ed35747d4cccb7e474f10026d679 P 03e839a9c2594f89b0e4f6cddff8a193: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:37.642302 26361 ts_tablet_manager.cc:1905] T 1076772bd258417e8199ca18605a24f2 P 1f5b3dd52a30437bb9dc4217ecee6741: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:37.642302 26362 ts_tablet_manager.cc:1905] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:37.643623 26328 tablet_bootstrap.cc:492] T c038ed35747d4cccb7e474f10026d679 P 03e839a9c2594f89b0e4f6cddff8a193: No bootstrap required, opened a new log
I20260810 07:25:37.643714 26328 ts_tablet_manager.cc:1397] T c038ed35747d4cccb7e474f10026d679 P 03e839a9c2594f89b0e4f6cddff8a193: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 07:25:37.644201 26328 raft_consensus.cc:348] T c038ed35747d4cccb7e474f10026d679 P 03e839a9c2594f89b0e4f6cddff8a193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:37.644338 26328 raft_consensus.cc:374] T c038ed35747d4cccb7e474f10026d679 P 03e839a9c2594f89b0e4f6cddff8a193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:37.644373 26328 raft_consensus.cc:729] T c038ed35747d4cccb7e474f10026d679 P 03e839a9c2594f89b0e4f6cddff8a193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e839a9c2594f89b0e4f6cddff8a193, State: Initialized, Role: FOLLOWER
I20260810 07:25:37.644516 26328 consensus_queue.cc:260] T c038ed35747d4cccb7e474f10026d679 P 03e839a9c2594f89b0e4f6cddff8a193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:37.644816 26328 ts_tablet_manager.cc:1428] T c038ed35747d4cccb7e474f10026d679 P 03e839a9c2594f89b0e4f6cddff8a193: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:37.645311 26361 ts_tablet_manager.cc:1918] T 1076772bd258417e8199ca18605a24f2 P 1f5b3dd52a30437bb9dc4217ecee6741: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:25:37.645478 26361 log.cc:1196] T 1076772bd258417e8199ca18605a24f2 P 1f5b3dd52a30437bb9dc4217ecee6741: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-0/wal/wals/1076772bd258417e8199ca18605a24f2
I20260810 07:25:37.645524 26362 ts_tablet_manager.cc:1918] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:25:37.645618 26362 log.cc:1196] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-2/wal/wals/1076772bd258417e8199ca18605a24f2
I20260810 07:25:37.645833 26361 ts_tablet_manager.cc:1939] T 1076772bd258417e8199ca18605a24f2 P 1f5b3dd52a30437bb9dc4217ecee6741: Deleting consensus metadata
I20260810 07:25:37.645931 26362 ts_tablet_manager.cc:1939] T 1076772bd258417e8199ca18605a24f2 P 96142be645164fd7887695f0986f8965: Deleting consensus metadata
I20260810 07:25:37.647120 25496 catalog_manager.cc:4841] TS 1f5b3dd52a30437bb9dc4217ecee6741 (127.12.157.193:42927): tablet 1076772bd258417e8199ca18605a24f2 (table test-table [id=326ad8e6ebf84d2da1779d0034c67ddf]) successfully deleted
I20260810 07:25:37.647707 25500 catalog_manager.cc:4841] TS 96142be645164fd7887695f0986f8965 (127.12.157.195:41691): tablet 1076772bd258417e8199ca18605a24f2 (table test-table [id=326ad8e6ebf84d2da1779d0034c67ddf]) successfully deleted
I20260810 07:25:37.648540 26325 tablet_bootstrap.cc:492] T c038ed35747d4cccb7e474f10026d679 P 96142be645164fd7887695f0986f8965: No bootstrap required, opened a new log
I20260810 07:25:37.648625 26325 ts_tablet_manager.cc:1397] T c038ed35747d4cccb7e474f10026d679 P 96142be645164fd7887695f0986f8965: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260810 07:25:37.649113 26325 raft_consensus.cc:348] T c038ed35747d4cccb7e474f10026d679 P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:37.649235 26325 raft_consensus.cc:374] T c038ed35747d4cccb7e474f10026d679 P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:37.649271 26325 raft_consensus.cc:729] T c038ed35747d4cccb7e474f10026d679 P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96142be645164fd7887695f0986f8965, State: Initialized, Role: FOLLOWER
I20260810 07:25:37.649521 26325 consensus_queue.cc:260] T c038ed35747d4cccb7e474f10026d679 P 96142be645164fd7887695f0986f8965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:37.649863 26325 ts_tablet_manager.cc:1428] T c038ed35747d4cccb7e474f10026d679 P 96142be645164fd7887695f0986f8965: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 07:25:37.650544 26331 tablet_bootstrap.cc:492] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741: No bootstrap required, opened a new log
I20260810 07:25:37.650625 26331 ts_tablet_manager.cc:1397] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20260810 07:25:37.651243 26331 raft_consensus.cc:348] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:37.651376 26331 raft_consensus.cc:374] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:37.651417 26331 raft_consensus.cc:729] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f5b3dd52a30437bb9dc4217ecee6741, State: Initialized, Role: FOLLOWER
I20260810 07:25:37.651556 26331 consensus_queue.cc:260] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:37.651872 26331 ts_tablet_manager.cc:1428] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260810 07:25:37.656054 26059 tablet.cc:2369] T c038ed35747d4cccb7e474f10026d679 P 03e839a9c2594f89b0e4f6cddff8a193: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 07:25:37.669950 25841 tablet.cc:2369] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:25:37.776811 26340 raft_consensus.cc:482] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:25:37.777032 26340 raft_consensus.cc:504] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:37.778527 26340 leader_election.cc:288] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03e839a9c2594f89b0e4f6cddff8a193 (127.12.157.194:38095), 96142be645164fd7887695f0986f8965 (127.12.157.195:41691)
I20260810 07:25:37.783347 26377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:37.778804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:41691 (local address 127.12.157.193:57713)
0810 07:25:37.779156 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:37.779171 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:37.779294 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:37.779661 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:37.779665 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:37.779687 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:37.779889 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:37.779897 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.781055 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.781058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:37.781980 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:37.781990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.782855 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.782858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:37.782981 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:37.783011 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 07:25:37.783095 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 07:25:37.783169 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":102,"spinlock_wait_cycles":22144,"thread_start_us":128,"threads_started":1}
I20260810 07:25:37.783389 26322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:37.779287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:57713 (local address 127.12.157.195:41691)
0810 07:25:37.779411 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:37.779416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:37.779431 (+ 15us) server_negotiation.cc:407] Connection header received
0810 07:25:37.779495 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:37.779498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:37.779546 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:37.779634 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:37.780026 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.780845 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.782132 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.782703 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.782827 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:37.782885 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:37.783148 (+ 263us) server_negotiation.cc:299] Negotiation successful
0810 07:25:37.783237 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 07:25:37.783771 26324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:37.778858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:34119 (local address 127.12.157.194:38095)
0810 07:25:37.778998 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:37.779004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:37.779201 (+ 197us) server_negotiation.cc:407] Connection header received
0810 07:25:37.779575 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:37.779579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:37.779659 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:37.779735 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:37.780118 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.780859 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.782445 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.783035 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.783177 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:37.783224 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:37.783578 (+ 354us) server_negotiation.cc:299] Negotiation successful
0810 07:25:37.783671 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 07:25:37.784255 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:37.778765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:38095 (local address 127.12.157.193:34119)
0810 07:25:37.779106 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:37.779122 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:37.779294 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:37.779770 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:37.779773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:37.779792 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:37.779961 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:37.779967 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.780996 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.780999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:37.782296 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:37.782308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.783305 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.783309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:37.783408 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:37.783441 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 07:25:37.784010 (+ 569us) client_negotiation.cc:240] Negotiation successful
0810 07:25:37.784074 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":38,"spinlock_wait_cycles":15744,"thread_start_us":74,"threads_started":1}
I20260810 07:25:37.784719 26207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c038ed35747d4cccb7e474f10026d679" candidate_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96142be645164fd7887695f0986f8965" is_pre_election: true
I20260810 07:25:37.784750 25982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c038ed35747d4cccb7e474f10026d679" candidate_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e839a9c2594f89b0e4f6cddff8a193" is_pre_election: true
I20260810 07:25:37.784933 25982 raft_consensus.cc:2454] T c038ed35747d4cccb7e474f10026d679 P 03e839a9c2594f89b0e4f6cddff8a193 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f5b3dd52a30437bb9dc4217ecee6741 in term 0.
I20260810 07:25:37.784996 26207 raft_consensus.cc:2454] T c038ed35747d4cccb7e474f10026d679 P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f5b3dd52a30437bb9dc4217ecee6741 in term 0.
I20260810 07:25:37.785408 25673 leader_election.cc:302] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 03e839a9c2594f89b0e4f6cddff8a193, 1f5b3dd52a30437bb9dc4217ecee6741, 96142be645164fd7887695f0986f8965; no voters:
I20260810 07:25:37.785751 26340 raft_consensus.cc:2781] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:25:37.785871 26340 raft_consensus.cc:482] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:25:37.785928 26340 raft_consensus.cc:3037] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:37.787016 26340 raft_consensus.cc:504] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:37.787467 26340 leader_election.cc:288] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741 [CANDIDATE]: Term 1 election: Requested vote from peers 03e839a9c2594f89b0e4f6cddff8a193 (127.12.157.194:38095), 96142be645164fd7887695f0986f8965 (127.12.157.195:41691)
I20260810 07:25:37.787856 25982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c038ed35747d4cccb7e474f10026d679" candidate_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e839a9c2594f89b0e4f6cddff8a193"
I20260810 07:25:37.787878 26207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c038ed35747d4cccb7e474f10026d679" candidate_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96142be645164fd7887695f0986f8965"
I20260810 07:25:37.787986 26207 raft_consensus.cc:3037] T c038ed35747d4cccb7e474f10026d679 P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:37.788017 25982 raft_consensus.cc:3037] T c038ed35747d4cccb7e474f10026d679 P 03e839a9c2594f89b0e4f6cddff8a193 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:37.789088 26207 raft_consensus.cc:2454] T c038ed35747d4cccb7e474f10026d679 P 96142be645164fd7887695f0986f8965 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f5b3dd52a30437bb9dc4217ecee6741 in term 1.
I20260810 07:25:37.789088 25982 raft_consensus.cc:2454] T c038ed35747d4cccb7e474f10026d679 P 03e839a9c2594f89b0e4f6cddff8a193 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f5b3dd52a30437bb9dc4217ecee6741 in term 1.
I20260810 07:25:37.789426 25673 leader_election.cc:302] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e839a9c2594f89b0e4f6cddff8a193, 1f5b3dd52a30437bb9dc4217ecee6741; no voters:
I20260810 07:25:37.789709 26340 raft_consensus.cc:2781] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:37.789978 26340 raft_consensus.cc:686] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 1 LEADER]: Becoming Leader. State: Replica: 1f5b3dd52a30437bb9dc4217ecee6741, State: Running, Role: LEADER
I20260810 07:25:37.790268 26340 consensus_queue.cc:237] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:37.793344 25519 catalog_manager.cc:5495] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f5b3dd52a30437bb9dc4217ecee6741 (127.12.157.193). New cstate: current_term: 1 leader_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } health_report { overall_health: UNKNOWN } } }
Aug 10 07:25:37 dist-test-slave-ffrd krb5kdc[25443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346736, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.254@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM
I20260810 07:25:37.875469 26390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:37.866838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:41615 (local address 127.0.0.1:39296)
0810 07:25:37.867209 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:37.867224 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:37.867335 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:37.867647 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:37.867649 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:37.868042 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:37.868143 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:37.868149 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.868998 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.869004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:37.869562 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:37.869573 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.870067 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.870070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:37.870135 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:37.870753 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:37.870766 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:37.872192 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:37.873940 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:37.873949 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:37.873953 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:37.874201 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:37.874505 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:37.874507 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:37.874509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:37.874590 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:37.874982 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:37.874987 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:37.875096 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 07:25:37.875205 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 07:25:37.875320 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":145,"threads_started":1}
I20260810 07:25:37.875540 25601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:37.866880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39296 (local address 127.12.157.254:41615)
0810 07:25:37.867036 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:37.867041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:37.867315 (+ 274us) server_negotiation.cc:407] Connection header received
0810 07:25:37.867465 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:37.867468 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:37.867514 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:37.867594 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:37.868281 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.868830 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.869682 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:37.869926 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:37.870009 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:37.872329 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:37.872332 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:37.872334 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:37.872362 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:37.873833 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:37.874316 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:37.874319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:37.874320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:37.874409 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:37.874682 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:37.874685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:37.874686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:37.874871 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:37.874952 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:37.875299 (+ 347us) server_negotiation.cc:299] Negotiation successful
0810 07:25:37.875401 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 07:25:37.879854 25519 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:39296:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 07:25:37.880604 25519 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:25:37.885768 25742 tablet_service.cc:1467] Processing CreateTablet for tablet 36ef19dba65c40bd9636ff33c76f1a7f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8137cbbe152c4d73854c68e8aaccad88]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 07:25:37.885768 25957 tablet_service.cc:1467] Processing CreateTablet for tablet 36ef19dba65c40bd9636ff33c76f1a7f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8137cbbe152c4d73854c68e8aaccad88]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 07:25:37.886000 26177 tablet_service.cc:1467] Processing CreateTablet for tablet 36ef19dba65c40bd9636ff33c76f1a7f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8137cbbe152c4d73854c68e8aaccad88]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 07:25:37.886184 25957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36ef19dba65c40bd9636ff33c76f1a7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:37.886183 25742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36ef19dba65c40bd9636ff33c76f1a7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:37.886438 26177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36ef19dba65c40bd9636ff33c76f1a7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:37.889235 26331 tablet_bootstrap.cc:492] T 36ef19dba65c40bd9636ff33c76f1a7f P 1f5b3dd52a30437bb9dc4217ecee6741: Bootstrap starting.
I20260810 07:25:37.889305 26325 tablet_bootstrap.cc:492] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965: Bootstrap starting.
I20260810 07:25:37.889660 26328 tablet_bootstrap.cc:492] T 36ef19dba65c40bd9636ff33c76f1a7f P 03e839a9c2594f89b0e4f6cddff8a193: Bootstrap starting.
I20260810 07:25:37.890280 26331 tablet_bootstrap.cc:654] T 36ef19dba65c40bd9636ff33c76f1a7f P 1f5b3dd52a30437bb9dc4217ecee6741: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:37.890362 26325 tablet_bootstrap.cc:654] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:37.890720 26328 tablet_bootstrap.cc:654] T 36ef19dba65c40bd9636ff33c76f1a7f P 03e839a9c2594f89b0e4f6cddff8a193: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:37.891492 26331 tablet_bootstrap.cc:492] T 36ef19dba65c40bd9636ff33c76f1a7f P 1f5b3dd52a30437bb9dc4217ecee6741: No bootstrap required, opened a new log
I20260810 07:25:37.891593 26331 ts_tablet_manager.cc:1397] T 36ef19dba65c40bd9636ff33c76f1a7f P 1f5b3dd52a30437bb9dc4217ecee6741: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 07:25:37.892083 26328 tablet_bootstrap.cc:492] T 36ef19dba65c40bd9636ff33c76f1a7f P 03e839a9c2594f89b0e4f6cddff8a193: No bootstrap required, opened a new log
I20260810 07:25:37.892189 26328 ts_tablet_manager.cc:1397] T 36ef19dba65c40bd9636ff33c76f1a7f P 03e839a9c2594f89b0e4f6cddff8a193: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 07:25:37.892102 26331 raft_consensus.cc:348] T 36ef19dba65c40bd9636ff33c76f1a7f P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:37.892241 26331 raft_consensus.cc:374] T 36ef19dba65c40bd9636ff33c76f1a7f P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:37.892283 26331 raft_consensus.cc:729] T 36ef19dba65c40bd9636ff33c76f1a7f P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f5b3dd52a30437bb9dc4217ecee6741, State: Initialized, Role: FOLLOWER
I20260810 07:25:37.892419 26325 tablet_bootstrap.cc:492] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965: No bootstrap required, opened a new log
I20260810 07:25:37.892429 26331 consensus_queue.cc:260] T 36ef19dba65c40bd9636ff33c76f1a7f P 1f5b3dd52a30437bb9dc4217ecee6741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:37.892513 26325 ts_tablet_manager.cc:1397] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 07:25:37.892691 26331 ts_tablet_manager.cc:1428] T 36ef19dba65c40bd9636ff33c76f1a7f P 1f5b3dd52a30437bb9dc4217ecee6741: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:37.892686 26328 raft_consensus.cc:348] T 36ef19dba65c40bd9636ff33c76f1a7f P 03e839a9c2594f89b0e4f6cddff8a193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:37.892827 26328 raft_consensus.cc:374] T 36ef19dba65c40bd9636ff33c76f1a7f P 03e839a9c2594f89b0e4f6cddff8a193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:37.892876 26328 raft_consensus.cc:729] T 36ef19dba65c40bd9636ff33c76f1a7f P 03e839a9c2594f89b0e4f6cddff8a193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e839a9c2594f89b0e4f6cddff8a193, State: Initialized, Role: FOLLOWER
I20260810 07:25:37.893014 26325 raft_consensus.cc:348] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:37.893064 26328 consensus_queue.cc:260] T 36ef19dba65c40bd9636ff33c76f1a7f P 03e839a9c2594f89b0e4f6cddff8a193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:37.893138 26325 raft_consensus.cc:374] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:37.893187 26325 raft_consensus.cc:729] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96142be645164fd7887695f0986f8965, State: Initialized, Role: FOLLOWER
I20260810 07:25:37.893316 26340 tablet_replica.cc:439] T 36ef19dba65c40bd9636ff33c76f1a7f P 1f5b3dd52a30437bb9dc4217ecee6741: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } } }
I20260810 07:25:37.893337 26325 consensus_queue.cc:260] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:37.893466 26340 tablet_replica.cc:442] T 36ef19dba65c40bd9636ff33c76f1a7f P 1f5b3dd52a30437bb9dc4217ecee6741: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:25:37.893649 26325 ts_tablet_manager.cc:1428] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 07:25:37.893760 26328 ts_tablet_manager.cc:1428] T 36ef19dba65c40bd9636ff33c76f1a7f P 03e839a9c2594f89b0e4f6cddff8a193: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 07:25:37.893740 26342 tablet_replica.cc:439] T 36ef19dba65c40bd9636ff33c76f1a7f P 03e839a9c2594f89b0e4f6cddff8a193: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } } }
I20260810 07:25:37.893898 26342 tablet_replica.cc:442] T 36ef19dba65c40bd9636ff33c76f1a7f P 03e839a9c2594f89b0e4f6cddff8a193: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:25:37.894871 26341 tablet_replica.cc:439] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } } }
I20260810 07:25:37.895037 26341 tablet_replica.cc:442] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965: This TxnStatusTablet replica's current role is: FOLLOWER
W20260810 07:25:37.895246 26282 tablet.cc:2369] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:25:38.033396 26341 raft_consensus.cc:482] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:25:38.033624 26341 raft_consensus.cc:504] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:38.034152 26341 leader_election.cc:288] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f5b3dd52a30437bb9dc4217ecee6741 (127.12.157.193:42927), 03e839a9c2594f89b0e4f6cddff8a193 (127.12.157.194:38095)
I20260810 07:25:38.034727 25982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ef19dba65c40bd9636ff33c76f1a7f" candidate_uuid: "96142be645164fd7887695f0986f8965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e839a9c2594f89b0e4f6cddff8a193" is_pre_election: true
I20260810 07:25:38.034729 25766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ef19dba65c40bd9636ff33c76f1a7f" candidate_uuid: "96142be645164fd7887695f0986f8965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" is_pre_election: true
I20260810 07:25:38.034866 25982 raft_consensus.cc:2454] T 36ef19dba65c40bd9636ff33c76f1a7f P 03e839a9c2594f89b0e4f6cddff8a193 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96142be645164fd7887695f0986f8965 in term 0.
I20260810 07:25:38.034987 25766 raft_consensus.cc:2454] T 36ef19dba65c40bd9636ff33c76f1a7f P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96142be645164fd7887695f0986f8965 in term 0.
I20260810 07:25:38.035254 26099 leader_election.cc:302] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e839a9c2594f89b0e4f6cddff8a193, 96142be645164fd7887695f0986f8965; no voters:
I20260810 07:25:38.035478 26341 raft_consensus.cc:2781] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:25:38.035539 26341 raft_consensus.cc:482] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:25:38.035597 26341 raft_consensus.cc:3037] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:38.036680 26341 raft_consensus.cc:504] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:38.037144 26341 leader_election.cc:288] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965 [CANDIDATE]: Term 1 election: Requested vote from peers 1f5b3dd52a30437bb9dc4217ecee6741 (127.12.157.193:42927), 03e839a9c2594f89b0e4f6cddff8a193 (127.12.157.194:38095)
I20260810 07:25:38.037528 25766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ef19dba65c40bd9636ff33c76f1a7f" candidate_uuid: "96142be645164fd7887695f0986f8965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f5b3dd52a30437bb9dc4217ecee6741"
I20260810 07:25:38.037575 25982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ef19dba65c40bd9636ff33c76f1a7f" candidate_uuid: "96142be645164fd7887695f0986f8965" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e839a9c2594f89b0e4f6cddff8a193"
I20260810 07:25:38.037667 25766 raft_consensus.cc:3037] T 36ef19dba65c40bd9636ff33c76f1a7f P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:38.037746 25982 raft_consensus.cc:3037] T 36ef19dba65c40bd9636ff33c76f1a7f P 03e839a9c2594f89b0e4f6cddff8a193 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:38.038952 25766 raft_consensus.cc:2454] T 36ef19dba65c40bd9636ff33c76f1a7f P 1f5b3dd52a30437bb9dc4217ecee6741 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96142be645164fd7887695f0986f8965 in term 1.
I20260810 07:25:38.038964 25982 raft_consensus.cc:2454] T 36ef19dba65c40bd9636ff33c76f1a7f P 03e839a9c2594f89b0e4f6cddff8a193 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96142be645164fd7887695f0986f8965 in term 1.
I20260810 07:25:38.039350 26100 leader_election.cc:302] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f5b3dd52a30437bb9dc4217ecee6741, 96142be645164fd7887695f0986f8965; no voters:
I20260810 07:25:38.039676 26341 raft_consensus.cc:2781] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:38.039805 26341 raft_consensus.cc:686] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965 [term 1 LEADER]: Becoming Leader. State: Replica: 96142be645164fd7887695f0986f8965, State: Running, Role: LEADER
I20260810 07:25:38.040006 26341 consensus_queue.cc:237] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:38.041105 26344 tablet_replica.cc:439] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965: TxnStatusTablet state changed. Reason: New leader 96142be645164fd7887695f0986f8965. Latest consensus state: current_term: 1 leader_uuid: "96142be645164fd7887695f0986f8965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } } }
I20260810 07:25:38.041246 26344 tablet_replica.cc:442] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965: This TxnStatusTablet replica's current role is: LEADER
I20260810 07:25:38.041621 26397 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 07:25:38.041808 26397 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 07:25:38.042039 25522 catalog_manager.cc:5495] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96142be645164fd7887695f0986f8965 (127.12.157.195). New cstate: current_term: 1 leader_uuid: "96142be645164fd7887695f0986f8965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } health_report { overall_health: HEALTHY } } }
I20260810 07:25:38.087157 26390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:38.083536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:41691 (local address 127.0.0.1:58452)
0810 07:25:38.083818 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:38.083835 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:38.083963 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:38.084272 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:38.084275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:38.084282 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 07:25:38.084418 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:38.084424 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.085251 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.085255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:38.085894 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:38.085903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.086374 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.086378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:38.086483 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:38.086529 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 07:25:38.086853 (+ 324us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 07:25:38.086872 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 07:25:38.086941 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 07:25:38.087026 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260810 07:25:38.087157 26322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:38.083674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58452 (local address 127.12.157.195:41691)
0810 07:25:38.083825 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:38.083831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:38.083887 (+ 56us) server_negotiation.cc:407] Connection header received
0810 07:25:38.084099 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:38.084102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:38.084157 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:38.084246 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:25:38.084543 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.085137 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.086027 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.086250 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.086366 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:38.086679 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:25:38.086761 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:25:38.086846 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:38.086952 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 07:25:38.087007 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 07:25:38.089998 25766 raft_consensus.cc:1261] T 36ef19dba65c40bd9636ff33c76f1a7f P 1f5b3dd52a30437bb9dc4217ecee6741 [term 1 FOLLOWER]: Refusing update from remote peer 96142be645164fd7887695f0986f8965: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:38.090132 25982 raft_consensus.cc:1261] T 36ef19dba65c40bd9636ff33c76f1a7f P 03e839a9c2594f89b0e4f6cddff8a193 [term 1 FOLLOWER]: Refusing update from remote peer 96142be645164fd7887695f0986f8965: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:38.090581 26344 consensus_queue.cc:1035] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:38.090802 26341 consensus_queue.cc:1035] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:38.092252 26340 tablet_replica.cc:439] T 36ef19dba65c40bd9636ff33c76f1a7f P 1f5b3dd52a30437bb9dc4217ecee6741: TxnStatusTablet state changed. Reason: New leader 96142be645164fd7887695f0986f8965. Latest consensus state: current_term: 1 leader_uuid: "96142be645164fd7887695f0986f8965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } } }
I20260810 07:25:38.092290 26342 tablet_replica.cc:439] T 36ef19dba65c40bd9636ff33c76f1a7f P 03e839a9c2594f89b0e4f6cddff8a193: TxnStatusTablet state changed. Reason: New leader 96142be645164fd7887695f0986f8965. Latest consensus state: current_term: 1 leader_uuid: "96142be645164fd7887695f0986f8965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } } }
I20260810 07:25:38.092379 26340 tablet_replica.cc:442] T 36ef19dba65c40bd9636ff33c76f1a7f P 1f5b3dd52a30437bb9dc4217ecee6741: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:25:38.092406 26342 tablet_replica.cc:442] T 36ef19dba65c40bd9636ff33c76f1a7f P 03e839a9c2594f89b0e4f6cddff8a193: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:25:38.093750 26341 tablet_replica.cc:439] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "96142be645164fd7887695f0986f8965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } } }
I20260810 07:25:38.093868 26341 tablet_replica.cc:442] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965: This TxnStatusTablet replica's current role is: LEADER
I20260810 07:25:38.094152 26341 tablet_replica.cc:439] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "96142be645164fd7887695f0986f8965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } } }
I20260810 07:25:38.094233 26341 tablet_replica.cc:442] T 36ef19dba65c40bd9636ff33c76f1a7f P 96142be645164fd7887695f0986f8965: This TxnStatusTablet replica's current role is: LEADER
I20260810 07:25:38.094304 26347 mvcc.cc:205] Tried to move back new op lower bound from 7316876239211868160 to 7316876239013724160. Current Snapshot: MvccSnapshot[applied={T|T < 7316876239211868160}]
I20260810 07:25:38.094844 26342 tablet_replica.cc:439] T 36ef19dba65c40bd9636ff33c76f1a7f P 03e839a9c2594f89b0e4f6cddff8a193: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "96142be645164fd7887695f0986f8965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } } }
I20260810 07:25:38.094966 26342 tablet_replica.cc:442] T 36ef19dba65c40bd9636ff33c76f1a7f P 03e839a9c2594f89b0e4f6cddff8a193: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:25:38.096099 26349 mvcc.cc:205] Tried to move back new op lower bound from 7316876239211868160 to 7316876239013724160. Current Snapshot: MvccSnapshot[applied={T|T < 7316876239211868160}]
I20260810 07:25:38.097078 26340 tablet_replica.cc:439] T 36ef19dba65c40bd9636ff33c76f1a7f P 1f5b3dd52a30437bb9dc4217ecee6741: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "96142be645164fd7887695f0986f8965" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } } }
I20260810 07:25:38.097206 26340 tablet_replica.cc:442] T 36ef19dba65c40bd9636ff33c76f1a7f P 1f5b3dd52a30437bb9dc4217ecee6741: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 07:25:38.104213 26407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:38.100310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41504 (local address 127.12.157.193:42927)
0810 07:25:38.100616 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:38.100624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:38.100638 (+ 14us) server_negotiation.cc:407] Connection header received
0810 07:25:38.100767 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:38.100770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:38.100839 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:38.100982 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:25:38.101252 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.101774 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.102652 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.102903 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.103019 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:38.103302 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:25:38.103760 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:25:38.103901 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:38.103968 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 07:25:38.104022 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":118,"threads_started":1}
I20260810 07:25:38.115831 26322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:38.112098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:54233 (local address 127.12.157.195:41691)
0810 07:25:38.112233 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:38.112236 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:38.112250 (+ 14us) server_negotiation.cc:407] Connection header received
0810 07:25:38.112374 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:38.112377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:38.112424 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:38.112488 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:25:38.112909 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.113506 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.114739 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.114980 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.115058 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:38.115407 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:25:38.115475 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:25:38.115525 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:38.115655 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 07:25:38.115704 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 07:25:38.115898 26413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:38.111815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:41691 (local address 127.12.157.193:54233)
0810 07:25:38.112148 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:38.112163 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:38.112263 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:38.112534 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:38.112537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:38.112544 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 07:25:38.112677 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:38.112683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.113630 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.113634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:38.114604 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:38.114616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.115099 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.115103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:38.115205 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:38.115262 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 07:25:38.115561 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 07:25:38.115568 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 07:25:38.115637 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 07:25:38.115719 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":101,"threads_started":1}
I20260810 07:25:38.123776 25982 raft_consensus.cc:1261] T c038ed35747d4cccb7e474f10026d679 P 03e839a9c2594f89b0e4f6cddff8a193 [term 1 FOLLOWER]: Refusing update from remote peer 1f5b3dd52a30437bb9dc4217ecee6741: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:38.124122 26207 raft_consensus.cc:1261] T c038ed35747d4cccb7e474f10026d679 P 96142be645164fd7887695f0986f8965 [term 1 FOLLOWER]: Refusing update from remote peer 1f5b3dd52a30437bb9dc4217ecee6741: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:38.124332 26378 consensus_queue.cc:1035] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:38.124616 26340 consensus_queue.cc:1035] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:38.147380 26407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:38.142990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:56219 (local address 127.12.157.193:42927)
0810 07:25:38.143131 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:38.143136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:38.143387 (+ 251us) server_negotiation.cc:407] Connection header received
0810 07:25:38.143627 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:38.143630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:38.143675 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:38.143738 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:25:38.144013 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.144607 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.146099 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.146438 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.146524 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:38.146972 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:25:38.147046 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:25:38.147113 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:38.147227 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 07:25:38.147273 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 07:25:38.147457 26424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:38.142914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:42927 (local address 127.12.157.195:56219)
0810 07:25:38.143296 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:38.143312 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:38.143467 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:38.143784 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:38.143787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:38.143797 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 07:25:38.143913 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:38.143920 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.144790 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.144794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:38.145949 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:38.145963 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.146643 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.146646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:38.146744 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:38.146799 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 07:25:38.147144 (+ 345us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 07:25:38.147150 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 07:25:38.147221 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 07:25:38.147304 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":141,"threads_started":1}
I20260810 07:25:38.177007 25522 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:38.177205 25522 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:38.179759 25522 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb0de2dd430e46078a8686e349202b26: Sending DeleteTablet for 3 replicas of tablet c038ed35747d4cccb7e474f10026d679
I20260810 07:25:38.180760 25957 tablet_service.cc:1514] Processing DeleteTablet for tablet c038ed35747d4cccb7e474f10026d679 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:38 UTC) from {username='oryx', principal='oryx/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:38940
I20260810 07:25:38.181128 26429 tablet_replica.cc:330] T c038ed35747d4cccb7e474f10026d679 P 03e839a9c2594f89b0e4f6cddff8a193: stopping tablet replica
I20260810 07:25:38.181265 26429 raft_consensus.cc:2229] T c038ed35747d4cccb7e474f10026d679 P 03e839a9c2594f89b0e4f6cddff8a193 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:25:38.181402 26429 raft_consensus.cc:2258] T c038ed35747d4cccb7e474f10026d679 P 03e839a9c2594f89b0e4f6cddff8a193 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:38.181545 25742 tablet_service.cc:1514] Processing DeleteTablet for tablet c038ed35747d4cccb7e474f10026d679 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:38 UTC) from {username='oryx', principal='oryx/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:41500
I20260810 07:25:38.181972 26177 tablet_service.cc:1514] Processing DeleteTablet for tablet c038ed35747d4cccb7e474f10026d679 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:38 UTC) from {username='oryx', principal='oryx/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:58434
I20260810 07:25:38.182108 26429 ts_tablet_manager.cc:1905] T c038ed35747d4cccb7e474f10026d679 P 03e839a9c2594f89b0e4f6cddff8a193: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:38.185328 26429 ts_tablet_manager.cc:1918] T c038ed35747d4cccb7e474f10026d679 P 03e839a9c2594f89b0e4f6cddff8a193: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 07:25:38.185420 26429 log.cc:1196] T c038ed35747d4cccb7e474f10026d679 P 03e839a9c2594f89b0e4f6cddff8a193: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-1/wal/wals/c038ed35747d4cccb7e474f10026d679
I20260810 07:25:38.185691 26429 ts_tablet_manager.cc:1939] T c038ed35747d4cccb7e474f10026d679 P 03e839a9c2594f89b0e4f6cddff8a193: Deleting consensus metadata
I20260810 07:25:38.186443 25495 catalog_manager.cc:4841] TS 03e839a9c2594f89b0e4f6cddff8a193 (127.12.157.194:38095): tablet c038ed35747d4cccb7e474f10026d679 (table test-table [id=d5dfbd6c2753401b9ee37d7257c704a1]) successfully deleted
I20260810 07:25:38.186779 26431 tablet_replica.cc:330] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741: stopping tablet replica
I20260810 07:25:38.186928 26431 raft_consensus.cc:2229] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:25:38.187275 26432 tablet_replica.cc:330] T c038ed35747d4cccb7e474f10026d679 P 96142be645164fd7887695f0986f8965: stopping tablet replica
I20260810 07:25:38.187278 26431 raft_consensus.cc:2258] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:38.187438 26432 raft_consensus.cc:2229] T c038ed35747d4cccb7e474f10026d679 P 96142be645164fd7887695f0986f8965 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:25:38.187595 26432 raft_consensus.cc:2258] T c038ed35747d4cccb7e474f10026d679 P 96142be645164fd7887695f0986f8965 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:38.187978 26431 ts_tablet_manager.cc:1905] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:38.188354 26432 ts_tablet_manager.cc:1905] T c038ed35747d4cccb7e474f10026d679 P 96142be645164fd7887695f0986f8965: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:38.191293 26431 ts_tablet_manager.cc:1918] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 07:25:38.191388 26431 log.cc:1196] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-0/wal/wals/c038ed35747d4cccb7e474f10026d679
I20260810 07:25:38.191687 26431 ts_tablet_manager.cc:1939] T c038ed35747d4cccb7e474f10026d679 P 1f5b3dd52a30437bb9dc4217ecee6741: Deleting consensus metadata
I20260810 07:25:38.191723 26432 ts_tablet_manager.cc:1918] T c038ed35747d4cccb7e474f10026d679 P 96142be645164fd7887695f0986f8965: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 07:25:38.191808 26432 log.cc:1196] T c038ed35747d4cccb7e474f10026d679 P 96142be645164fd7887695f0986f8965: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-2/wal/wals/c038ed35747d4cccb7e474f10026d679
I20260810 07:25:38.192086 26432 ts_tablet_manager.cc:1939] T c038ed35747d4cccb7e474f10026d679 P 96142be645164fd7887695f0986f8965: Deleting consensus metadata
I20260810 07:25:38.192479 25496 catalog_manager.cc:4841] TS 1f5b3dd52a30437bb9dc4217ecee6741 (127.12.157.193:42927): tablet c038ed35747d4cccb7e474f10026d679 (table test-table [id=d5dfbd6c2753401b9ee37d7257c704a1]) successfully deleted
I20260810 07:25:38.192811 25500 catalog_manager.cc:4841] TS 96142be645164fd7887695f0986f8965 (127.12.157.195:41691): tablet c038ed35747d4cccb7e474f10026d679 (table test-table [id=d5dfbd6c2753401b9ee37d7257c704a1]) successfully deleted
Aug 10 07:25:38 dist-test-slave-ffrd krb5kdc[25443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 07:25:38 dist-test-slave-ffrd krb5kdc[25443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:38 dist-test-slave-ffrd krb5kdc[25443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 07:25:38.220827 25601 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:38.212745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39300 (local address 127.12.157.254:41615)
0810 07:25:38.212879 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:38.212884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:38.213215 (+ 331us) server_negotiation.cc:407] Connection header received
0810 07:25:38.213349 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:38.213353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:38.213418 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:38.213522 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:38.214394 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.215231 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.216465 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.216825 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.216925 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:38.220680 (+ 3755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39300: BlockingRecv error: recv got EOF from 127.0.0.1:39300 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 07:25:38 dist-test-slave-ffrd krb5kdc[25443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM
I20260810 07:25:38.232035 25601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:38.222851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39310 (local address 127.12.157.254:41615)
0810 07:25:38.222955 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:38.222959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:38.223194 (+ 235us) server_negotiation.cc:407] Connection header received
0810 07:25:38.223292 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:38.223295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:38.223345 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:38.223423 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:38.224117 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.224910 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.225708 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.226062 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.226167 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:38.228692 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:38.228696 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:38.228699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:38.228726 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:38.230428 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:38.230854 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:38.230857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:38.230859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:38.230949 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:38.231218 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:38.231221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:38.231223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:38.231432 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:38.231570 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:38.231738 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 07:25:38.231849 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 07:25:38.234946 25522 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:25:38.235437 25522 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:25:38.240470 25742 tablet_service.cc:1467] Processing CreateTablet for tablet 4353f238e6f14ba5b5ea13bd41da3769 (DEFAULT_TABLE table=test-table [id=350662af4ac9425c8dd777b48ca9b9c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:38.240664 26177 tablet_service.cc:1467] Processing CreateTablet for tablet 4353f238e6f14ba5b5ea13bd41da3769 (DEFAULT_TABLE table=test-table [id=350662af4ac9425c8dd777b48ca9b9c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:38.240794 25742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4353f238e6f14ba5b5ea13bd41da3769. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:38.240979 26177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4353f238e6f14ba5b5ea13bd41da3769. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:38.243333 26325 tablet_bootstrap.cc:492] T 4353f238e6f14ba5b5ea13bd41da3769 P 96142be645164fd7887695f0986f8965: Bootstrap starting.
I20260810 07:25:38.243584 25957 tablet_service.cc:1467] Processing CreateTablet for tablet 4353f238e6f14ba5b5ea13bd41da3769 (DEFAULT_TABLE table=test-table [id=350662af4ac9425c8dd777b48ca9b9c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:38.243897 25957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4353f238e6f14ba5b5ea13bd41da3769. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:38.244215 26325 tablet_bootstrap.cc:654] T 4353f238e6f14ba5b5ea13bd41da3769 P 96142be645164fd7887695f0986f8965: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:38.244753 26331 tablet_bootstrap.cc:492] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741: Bootstrap starting.
I20260810 07:25:38.245700 26331 tablet_bootstrap.cc:654] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:38.245935 26328 tablet_bootstrap.cc:492] T 4353f238e6f14ba5b5ea13bd41da3769 P 03e839a9c2594f89b0e4f6cddff8a193: Bootstrap starting.
I20260810 07:25:38.246691 26325 tablet_bootstrap.cc:492] T 4353f238e6f14ba5b5ea13bd41da3769 P 96142be645164fd7887695f0986f8965: No bootstrap required, opened a new log
I20260810 07:25:38.246778 26325 ts_tablet_manager.cc:1397] T 4353f238e6f14ba5b5ea13bd41da3769 P 96142be645164fd7887695f0986f8965: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 07:25:38.246912 26328 tablet_bootstrap.cc:654] T 4353f238e6f14ba5b5ea13bd41da3769 P 03e839a9c2594f89b0e4f6cddff8a193: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:38.247175 26325 raft_consensus.cc:348] T 4353f238e6f14ba5b5ea13bd41da3769 P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:38.247296 26325 raft_consensus.cc:374] T 4353f238e6f14ba5b5ea13bd41da3769 P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:38.247332 26325 raft_consensus.cc:729] T 4353f238e6f14ba5b5ea13bd41da3769 P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96142be645164fd7887695f0986f8965, State: Initialized, Role: FOLLOWER
I20260810 07:25:38.247450 26325 consensus_queue.cc:260] T 4353f238e6f14ba5b5ea13bd41da3769 P 96142be645164fd7887695f0986f8965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:38.247715 26325 ts_tablet_manager.cc:1428] T 4353f238e6f14ba5b5ea13bd41da3769 P 96142be645164fd7887695f0986f8965: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:38.248631 26328 tablet_bootstrap.cc:492] T 4353f238e6f14ba5b5ea13bd41da3769 P 03e839a9c2594f89b0e4f6cddff8a193: No bootstrap required, opened a new log
I20260810 07:25:38.248718 26328 ts_tablet_manager.cc:1397] T 4353f238e6f14ba5b5ea13bd41da3769 P 03e839a9c2594f89b0e4f6cddff8a193: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 07:25:38.249029 26331 tablet_bootstrap.cc:492] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741: No bootstrap required, opened a new log
I20260810 07:25:38.249114 26331 ts_tablet_manager.cc:1397] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 07:25:38.249244 26328 raft_consensus.cc:348] T 4353f238e6f14ba5b5ea13bd41da3769 P 03e839a9c2594f89b0e4f6cddff8a193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:38.249351 26328 raft_consensus.cc:374] T 4353f238e6f14ba5b5ea13bd41da3769 P 03e839a9c2594f89b0e4f6cddff8a193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:38.249392 26328 raft_consensus.cc:729] T 4353f238e6f14ba5b5ea13bd41da3769 P 03e839a9c2594f89b0e4f6cddff8a193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e839a9c2594f89b0e4f6cddff8a193, State: Initialized, Role: FOLLOWER
I20260810 07:25:38.249553 26328 consensus_queue.cc:260] T 4353f238e6f14ba5b5ea13bd41da3769 P 03e839a9c2594f89b0e4f6cddff8a193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:38.249569 26331 raft_consensus.cc:348] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:38.249675 26331 raft_consensus.cc:374] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:38.249706 26331 raft_consensus.cc:729] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f5b3dd52a30437bb9dc4217ecee6741, State: Initialized, Role: FOLLOWER
I20260810 07:25:38.249814 26331 consensus_queue.cc:260] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:38.250053 26331 ts_tablet_manager.cc:1428] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:38.251457 26328 ts_tablet_manager.cc:1428] T 4353f238e6f14ba5b5ea13bd41da3769 P 03e839a9c2594f89b0e4f6cddff8a193: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260810 07:25:38.371829 26340 raft_consensus.cc:482] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:25:38.371963 26340 raft_consensus.cc:504] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:38.372430 26340 leader_election.cc:288] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03e839a9c2594f89b0e4f6cddff8a193 (127.12.157.194:38095), 96142be645164fd7887695f0986f8965 (127.12.157.195:41691)
I20260810 07:25:38.373001 25982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4353f238e6f14ba5b5ea13bd41da3769" candidate_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e839a9c2594f89b0e4f6cddff8a193" is_pre_election: true
I20260810 07:25:38.373001 26207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4353f238e6f14ba5b5ea13bd41da3769" candidate_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96142be645164fd7887695f0986f8965" is_pre_election: true
I20260810 07:25:38.373169 25982 raft_consensus.cc:2454] T 4353f238e6f14ba5b5ea13bd41da3769 P 03e839a9c2594f89b0e4f6cddff8a193 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f5b3dd52a30437bb9dc4217ecee6741 in term 0.
I20260810 07:25:38.373169 26207 raft_consensus.cc:2454] T 4353f238e6f14ba5b5ea13bd41da3769 P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f5b3dd52a30437bb9dc4217ecee6741 in term 0.
I20260810 07:25:38.373509 25679 leader_election.cc:302] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f5b3dd52a30437bb9dc4217ecee6741, 96142be645164fd7887695f0986f8965; no voters:
I20260810 07:25:38.373710 26340 raft_consensus.cc:2781] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:25:38.373787 26340 raft_consensus.cc:482] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:25:38.373843 26340 raft_consensus.cc:3037] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:38.374869 26340 raft_consensus.cc:504] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:38.375241 26340 leader_election.cc:288] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741 [CANDIDATE]: Term 1 election: Requested vote from peers 03e839a9c2594f89b0e4f6cddff8a193 (127.12.157.194:38095), 96142be645164fd7887695f0986f8965 (127.12.157.195:41691)
I20260810 07:25:38.375682 26207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4353f238e6f14ba5b5ea13bd41da3769" candidate_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96142be645164fd7887695f0986f8965"
I20260810 07:25:38.375682 25982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4353f238e6f14ba5b5ea13bd41da3769" candidate_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03e839a9c2594f89b0e4f6cddff8a193"
I20260810 07:25:38.375821 26207 raft_consensus.cc:3037] T 4353f238e6f14ba5b5ea13bd41da3769 P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:38.375821 25982 raft_consensus.cc:3037] T 4353f238e6f14ba5b5ea13bd41da3769 P 03e839a9c2594f89b0e4f6cddff8a193 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:38.377067 25982 raft_consensus.cc:2454] T 4353f238e6f14ba5b5ea13bd41da3769 P 03e839a9c2594f89b0e4f6cddff8a193 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f5b3dd52a30437bb9dc4217ecee6741 in term 1.
I20260810 07:25:38.377067 26207 raft_consensus.cc:2454] T 4353f238e6f14ba5b5ea13bd41da3769 P 96142be645164fd7887695f0986f8965 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f5b3dd52a30437bb9dc4217ecee6741 in term 1.
I20260810 07:25:38.377373 25673 leader_election.cc:302] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e839a9c2594f89b0e4f6cddff8a193, 1f5b3dd52a30437bb9dc4217ecee6741; no voters:
I20260810 07:25:38.377563 26340 raft_consensus.cc:2781] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:38.377667 26340 raft_consensus.cc:686] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 1 LEADER]: Becoming Leader. State: Replica: 1f5b3dd52a30437bb9dc4217ecee6741, State: Running, Role: LEADER
I20260810 07:25:38.377815 26340 consensus_queue.cc:237] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:38.379513 25522 catalog_manager.cc:5495] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f5b3dd52a30437bb9dc4217ecee6741 (127.12.157.193). New cstate: current_term: 1 leader_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } health_report { overall_health: UNKNOWN } } }
I20260810 07:25:38.407196 26407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:38.403651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41510 (local address 127.12.157.193:42927)
0810 07:25:38.403789 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:38.403793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:38.403808 (+ 15us) server_negotiation.cc:407] Connection header received
0810 07:25:38.403887 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:38.403890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:38.403937 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:38.404017 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:25:38.404236 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.405117 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.405998 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.406349 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.406475 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:38.406715 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:25:38.406804 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:25:38.406873 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:38.406993 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 07:25:38.407057 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 07:25:38.409302 26207 raft_consensus.cc:1261] T 4353f238e6f14ba5b5ea13bd41da3769 P 96142be645164fd7887695f0986f8965 [term 1 FOLLOWER]: Refusing update from remote peer 1f5b3dd52a30437bb9dc4217ecee6741: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:38.409309 25982 raft_consensus.cc:1261] T 4353f238e6f14ba5b5ea13bd41da3769 P 03e839a9c2594f89b0e4f6cddff8a193 [term 1 FOLLOWER]: Refusing update from remote peer 1f5b3dd52a30437bb9dc4217ecee6741: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:38.409736 26340 consensus_queue.cc:1035] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:38.409934 26378 consensus_queue.cc:1035] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:38.417819 25522 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:38.417955 25522 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:38.420176 25522 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb0de2dd430e46078a8686e349202b26: Sending DeleteTablet for 3 replicas of tablet 4353f238e6f14ba5b5ea13bd41da3769
I20260810 07:25:38.421060 25742 tablet_service.cc:1514] Processing DeleteTablet for tablet 4353f238e6f14ba5b5ea13bd41da3769 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:38 UTC) from {username='oryx', principal='oryx/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:41500
I20260810 07:25:38.421275 26431 tablet_replica.cc:330] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741: stopping tablet replica
I20260810 07:25:38.421810 25957 tablet_service.cc:1514] Processing DeleteTablet for tablet 4353f238e6f14ba5b5ea13bd41da3769 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:38 UTC) from {username='oryx', principal='oryx/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:38940
I20260810 07:25:38.421838 26431 raft_consensus.cc:2229] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:25:38.422039 26431 raft_consensus.cc:2258] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:38.422078 26429 tablet_replica.cc:330] T 4353f238e6f14ba5b5ea13bd41da3769 P 03e839a9c2594f89b0e4f6cddff8a193: stopping tablet replica
I20260810 07:25:38.422119 26177 tablet_service.cc:1514] Processing DeleteTablet for tablet 4353f238e6f14ba5b5ea13bd41da3769 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:38 UTC) from {username='oryx', principal='oryx/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:58434
I20260810 07:25:38.422194 26429 raft_consensus.cc:2229] T 4353f238e6f14ba5b5ea13bd41da3769 P 03e839a9c2594f89b0e4f6cddff8a193 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:25:38.422290 26432 tablet_replica.cc:330] T 4353f238e6f14ba5b5ea13bd41da3769 P 96142be645164fd7887695f0986f8965: stopping tablet replica
I20260810 07:25:38.422304 26429 raft_consensus.cc:2258] T 4353f238e6f14ba5b5ea13bd41da3769 P 03e839a9c2594f89b0e4f6cddff8a193 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:38.422412 26432 raft_consensus.cc:2229] T 4353f238e6f14ba5b5ea13bd41da3769 P 96142be645164fd7887695f0986f8965 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:25:38.422544 26432 raft_consensus.cc:2258] T 4353f238e6f14ba5b5ea13bd41da3769 P 96142be645164fd7887695f0986f8965 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:38.422807 26431 ts_tablet_manager.cc:1905] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:38.423009 25522 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 07:25:38.423214 26432 ts_tablet_manager.cc:1905] T 4353f238e6f14ba5b5ea13bd41da3769 P 96142be645164fd7887695f0986f8965: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 07:25:38.423578 25522 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:25:38.425011 26429 ts_tablet_manager.cc:1905] T 4353f238e6f14ba5b5ea13bd41da3769 P 03e839a9c2594f89b0e4f6cddff8a193: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:38.425585 26432 ts_tablet_manager.cc:1918] T 4353f238e6f14ba5b5ea13bd41da3769 P 96142be645164fd7887695f0986f8965: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:25:38.425669 26432 log.cc:1196] T 4353f238e6f14ba5b5ea13bd41da3769 P 96142be645164fd7887695f0986f8965: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-2/wal/wals/4353f238e6f14ba5b5ea13bd41da3769
I20260810 07:25:38.425714 26431 ts_tablet_manager.cc:1918] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:25:38.425788 26431 log.cc:1196] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-0/wal/wals/4353f238e6f14ba5b5ea13bd41da3769
I20260810 07:25:38.425956 26432 ts_tablet_manager.cc:1939] T 4353f238e6f14ba5b5ea13bd41da3769 P 96142be645164fd7887695f0986f8965: Deleting consensus metadata
I20260810 07:25:38.426054 26431 ts_tablet_manager.cc:1939] T 4353f238e6f14ba5b5ea13bd41da3769 P 1f5b3dd52a30437bb9dc4217ecee6741: Deleting consensus metadata
I20260810 07:25:38.426652 25500 catalog_manager.cc:4841] TS 96142be645164fd7887695f0986f8965 (127.12.157.195:41691): tablet 4353f238e6f14ba5b5ea13bd41da3769 (table test-table [id=350662af4ac9425c8dd777b48ca9b9c9]) successfully deleted
I20260810 07:25:38.426745 25496 catalog_manager.cc:4841] TS 1f5b3dd52a30437bb9dc4217ecee6741 (127.12.157.193:42927): tablet 4353f238e6f14ba5b5ea13bd41da3769 (table test-table [id=350662af4ac9425c8dd777b48ca9b9c9]) successfully deleted
I20260810 07:25:38.427484 26429 ts_tablet_manager.cc:1918] T 4353f238e6f14ba5b5ea13bd41da3769 P 03e839a9c2594f89b0e4f6cddff8a193: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:25:38.427577 26429 log.cc:1196] T 4353f238e6f14ba5b5ea13bd41da3769 P 03e839a9c2594f89b0e4f6cddff8a193: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-1/wal/wals/4353f238e6f14ba5b5ea13bd41da3769
I20260810 07:25:38.427858 26429 ts_tablet_manager.cc:1939] T 4353f238e6f14ba5b5ea13bd41da3769 P 03e839a9c2594f89b0e4f6cddff8a193: Deleting consensus metadata
I20260810 07:25:38.428485 25495 catalog_manager.cc:4841] TS 03e839a9c2594f89b0e4f6cddff8a193 (127.12.157.194:38095): tablet 4353f238e6f14ba5b5ea13bd41da3769 (table test-table [id=350662af4ac9425c8dd777b48ca9b9c9]) successfully deleted
I20260810 07:25:38.428514 25742 tablet_service.cc:1467] Processing CreateTablet for tablet 5dc80ae8e16943d9b2f5bd1cb4c9f09a (DEFAULT_TABLE table=test-table [id=2ba512ec64a449bd97acaab5fc2c6e89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:38.428771 25742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dc80ae8e16943d9b2f5bd1cb4c9f09a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:38.428784 25957 tablet_service.cc:1467] Processing CreateTablet for tablet 5dc80ae8e16943d9b2f5bd1cb4c9f09a (DEFAULT_TABLE table=test-table [id=2ba512ec64a449bd97acaab5fc2c6e89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:38.428983 26177 tablet_service.cc:1467] Processing CreateTablet for tablet 5dc80ae8e16943d9b2f5bd1cb4c9f09a (DEFAULT_TABLE table=test-table [id=2ba512ec64a449bd97acaab5fc2c6e89]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:25:38.429044 25957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dc80ae8e16943d9b2f5bd1cb4c9f09a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:38.429217 26177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dc80ae8e16943d9b2f5bd1cb4c9f09a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:38.430757 26331 tablet_bootstrap.cc:492] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 1f5b3dd52a30437bb9dc4217ecee6741: Bootstrap starting.
I20260810 07:25:38.431226 26328 tablet_bootstrap.cc:492] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193: Bootstrap starting.
I20260810 07:25:38.431649 26331 tablet_bootstrap.cc:654] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 1f5b3dd52a30437bb9dc4217ecee6741: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:38.431857 26325 tablet_bootstrap.cc:492] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 96142be645164fd7887695f0986f8965: Bootstrap starting.
I20260810 07:25:38.432086 26328 tablet_bootstrap.cc:654] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:38.432667 26325 tablet_bootstrap.cc:654] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 96142be645164fd7887695f0986f8965: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:38.432880 26331 tablet_bootstrap.cc:492] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 1f5b3dd52a30437bb9dc4217ecee6741: No bootstrap required, opened a new log
I20260810 07:25:38.432994 26331 ts_tablet_manager.cc:1397] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 1f5b3dd52a30437bb9dc4217ecee6741: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 07:25:38.433411 26328 tablet_bootstrap.cc:492] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193: No bootstrap required, opened a new log
I20260810 07:25:38.433508 26328 ts_tablet_manager.cc:1397] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 07:25:38.433420 26331 raft_consensus.cc:348] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:38.433573 26331 raft_consensus.cc:374] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:38.433626 26331 raft_consensus.cc:729] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f5b3dd52a30437bb9dc4217ecee6741, State: Initialized, Role: FOLLOWER
I20260810 07:25:38.433787 26331 consensus_queue.cc:260] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 1f5b3dd52a30437bb9dc4217ecee6741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:38.433936 26325 tablet_bootstrap.cc:492] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 96142be645164fd7887695f0986f8965: No bootstrap required, opened a new log
I20260810 07:25:38.434022 26325 ts_tablet_manager.cc:1397] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 96142be645164fd7887695f0986f8965: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 07:25:38.433956 26328 raft_consensus.cc:348] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:38.434067 26328 raft_consensus.cc:374] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:38.434111 26328 raft_consensus.cc:729] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03e839a9c2594f89b0e4f6cddff8a193, State: Initialized, Role: FOLLOWER
I20260810 07:25:38.434149 26331 ts_tablet_manager.cc:1428] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 1f5b3dd52a30437bb9dc4217ecee6741: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:38.434239 26328 consensus_queue.cc:260] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:38.434427 26325 raft_consensus.cc:348] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:38.434513 26328 ts_tablet_manager.cc:1428] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:38.434549 26325 raft_consensus.cc:374] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:38.434592 26325 raft_consensus.cc:729] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96142be645164fd7887695f0986f8965, State: Initialized, Role: FOLLOWER
I20260810 07:25:38.434728 26325 consensus_queue.cc:260] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 96142be645164fd7887695f0986f8965 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:38.435002 26325 ts_tablet_manager.cc:1428] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 96142be645164fd7887695f0986f8965: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 07:25:38.599038 26342 raft_consensus.cc:482] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:25:38.599207 26342 raft_consensus.cc:504] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:38.600412 26342 leader_election.cc:288] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f5b3dd52a30437bb9dc4217ecee6741 (127.12.157.193:42927), 96142be645164fd7887695f0986f8965 (127.12.157.195:41691)
I20260810 07:25:38.604584 26322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:38.600805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:40107 (local address 127.12.157.195:41691)
0810 07:25:38.601028 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:38.601034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:38.601117 (+ 83us) server_negotiation.cc:407] Connection header received
0810 07:25:38.601271 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:38.601274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:38.601321 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:38.601415 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:38.601755 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.602244 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.603501 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.603891 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.604021 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:38.604079 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:38.604374 (+ 295us) server_negotiation.cc:299] Negotiation successful
0810 07:25:38.604453 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260810 07:25:38.604581 26407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:38.600727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:57375 (local address 127.12.157.193:42927)
0810 07:25:38.600884 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:38.600890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:38.601077 (+ 187us) server_negotiation.cc:407] Connection header received
0810 07:25:38.601254 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:38.601257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:38.601307 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:38.601371 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:25:38.601699 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.602249 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.603462 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.603895 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.604041 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:38.604102 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:38.604349 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 07:25:38.604439 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 07:25:38.604610 26472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:38.600607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:42927 (local address 127.12.157.194:57375)
0810 07:25:38.600958 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:38.600975 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:38.601122 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:38.601413 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:38.601416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:38.601433 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:38.601584 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:38.601590 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.602414 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.602417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:38.603323 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:38.603332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.604041 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.604045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:38.604218 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:38.604240 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 07:25:38.604302 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 07:25:38.604440 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":98,"spinlock_wait_cycles":21760,"thread_start_us":86,"threads_started":1}
I20260810 07:25:38.604621 26471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:38.600718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:41691 (local address 127.12.157.194:40107)
0810 07:25:38.600995 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:38.601009 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:38.601147 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:38.601444 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:38.601447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:38.601493 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:25:38.601611 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:38.601616 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.602390 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.602393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:38.603333 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:38.603350 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.604035 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.604039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:38.604224 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:38.604244 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 07:25:38.604337 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 07:25:38.604439 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":100,"spinlock_wait_cycles":3200,"thread_start_us":69,"threads_started":1}
I20260810 07:25:38.605060 25766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc80ae8e16943d9b2f5bd1cb4c9f09a" candidate_uuid: "03e839a9c2594f89b0e4f6cddff8a193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" is_pre_election: true
I20260810 07:25:38.605222 25766 raft_consensus.cc:2454] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03e839a9c2594f89b0e4f6cddff8a193 in term 0.
I20260810 07:25:38.605513 26207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc80ae8e16943d9b2f5bd1cb4c9f09a" candidate_uuid: "03e839a9c2594f89b0e4f6cddff8a193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96142be645164fd7887695f0986f8965" is_pre_election: true
I20260810 07:25:38.605664 26207 raft_consensus.cc:2454] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03e839a9c2594f89b0e4f6cddff8a193 in term 0.
I20260810 07:25:38.605620 25879 leader_election.cc:302] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e839a9c2594f89b0e4f6cddff8a193, 1f5b3dd52a30437bb9dc4217ecee6741; no voters:
I20260810 07:25:38.605933 26342 raft_consensus.cc:2781] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:25:38.606017 26342 raft_consensus.cc:482] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:25:38.606074 26342 raft_consensus.cc:3037] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:38.607084 26342 raft_consensus.cc:504] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:38.607437 26342 leader_election.cc:288] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193 [CANDIDATE]: Term 1 election: Requested vote from peers 1f5b3dd52a30437bb9dc4217ecee6741 (127.12.157.193:42927), 96142be645164fd7887695f0986f8965 (127.12.157.195:41691)
I20260810 07:25:38.607780 25766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc80ae8e16943d9b2f5bd1cb4c9f09a" candidate_uuid: "03e839a9c2594f89b0e4f6cddff8a193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f5b3dd52a30437bb9dc4217ecee6741"
I20260810 07:25:38.607812 26207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc80ae8e16943d9b2f5bd1cb4c9f09a" candidate_uuid: "03e839a9c2594f89b0e4f6cddff8a193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96142be645164fd7887695f0986f8965"
I20260810 07:25:38.607883 25766 raft_consensus.cc:3037] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 1f5b3dd52a30437bb9dc4217ecee6741 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:38.607959 26207 raft_consensus.cc:3037] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 96142be645164fd7887695f0986f8965 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:38.608841 25766 raft_consensus.cc:2454] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 1f5b3dd52a30437bb9dc4217ecee6741 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03e839a9c2594f89b0e4f6cddff8a193 in term 1.
I20260810 07:25:38.608999 26207 raft_consensus.cc:2454] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 96142be645164fd7887695f0986f8965 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03e839a9c2594f89b0e4f6cddff8a193 in term 1.
I20260810 07:25:38.609144 25879 leader_election.cc:302] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03e839a9c2594f89b0e4f6cddff8a193, 1f5b3dd52a30437bb9dc4217ecee6741; no voters:
I20260810 07:25:38.609390 26342 raft_consensus.cc:2781] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:38.609561 26342 raft_consensus.cc:686] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193 [term 1 LEADER]: Becoming Leader. State: Replica: 03e839a9c2594f89b0e4f6cddff8a193, State: Running, Role: LEADER
I20260810 07:25:38.609812 26342 consensus_queue.cc:237] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } }
I20260810 07:25:38.612296 25522 catalog_manager.cc:5495] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03e839a9c2594f89b0e4f6cddff8a193 (127.12.157.194). New cstate: current_term: 1 leader_uuid: "03e839a9c2594f89b0e4f6cddff8a193" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03e839a9c2594f89b0e4f6cddff8a193" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 } health_report { overall_health: UNKNOWN } } }
I20260810 07:25:38.634729 26475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:38.630516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38948 (local address 127.12.157.194:38095)
0810 07:25:38.630923 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:38.630933 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:38.630960 (+ 27us) server_negotiation.cc:407] Connection header received
0810 07:25:38.631044 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:38.631046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:38.631100 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:38.631205 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:25:38.631480 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.632134 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.633075 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.633390 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.633512 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:38.633813 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:25:38.634302 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:25:38.634441 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:38.634501 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 07:25:38.634557 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":118,"threads_started":1}
I20260810 07:25:38.645357 26479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:38.641651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:41691 (local address 127.12.157.194:60575)
0810 07:25:38.642053 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:38.642065 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:38.642195 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:38.642471 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:38.642474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:38.642482 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 07:25:38.642611 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:38.642616 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.643345 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.643349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:38.644006 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:38.644015 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.644483 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.644487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:38.644592 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:38.644639 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 07:25:38.645049 (+ 410us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 07:25:38.645055 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 07:25:38.645109 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 07:25:38.645219 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":174,"threads_started":1}
I20260810 07:25:38.645381 26322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:38.641821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:60575 (local address 127.12.157.195:41691)
0810 07:25:38.641991 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:38.641996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:38.642143 (+ 147us) server_negotiation.cc:407] Connection header received
0810 07:25:38.642308 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:38.642311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:38.642367 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:38.642449 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:25:38.642719 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.643230 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.644138 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.644363 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.644470 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:38.644788 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:25:38.644869 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:25:38.644991 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:38.645209 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 07:25:38.645277 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 07:25:38.651904 25766 raft_consensus.cc:1261] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 1f5b3dd52a30437bb9dc4217ecee6741 [term 1 FOLLOWER]: Refusing update from remote peer 03e839a9c2594f89b0e4f6cddff8a193: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:38.651989 26207 raft_consensus.cc:1261] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 96142be645164fd7887695f0986f8965 [term 1 FOLLOWER]: Refusing update from remote peer 03e839a9c2594f89b0e4f6cddff8a193: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:25:38.652483 26342 consensus_queue.cc:1035] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f5b3dd52a30437bb9dc4217ecee6741" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:38.653141 26473 consensus_queue.cc:1035] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96142be645164fd7887695f0986f8965" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:25:38.671906 26475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:38.667948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:33421 (local address 127.12.157.194:38095)
0810 07:25:38.668093 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:38.668098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:38.668241 (+ 143us) server_negotiation.cc:407] Connection header received
0810 07:25:38.668331 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:38.668333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:38.668530 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:38.668613 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:25:38.668926 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.669712 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.670804 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.671041 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.671155 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:38.671430 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:25:38.671517 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:25:38.671602 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:38.671694 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 07:25:38.671757 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 07:25:38.671921 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:38.667866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:38095 (local address 127.12.157.195:33421)
0810 07:25:38.668127 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:38.668143 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:38.668247 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:38.668638 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:38.668641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:38.668653 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 07:25:38.668786 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:38.668792 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.669854 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.669861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:38.670680 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:38.670691 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:38.671168 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:38.671172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:38.671264 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:38.671300 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 07:25:38.671607 (+ 307us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 07:25:38.671617 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 07:25:38.671705 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 07:25:38.671787 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":84,"threads_started":1}
I20260810 07:25:38.700860 25519 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:38.701042 25519 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:25:38.703392 25519 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb0de2dd430e46078a8686e349202b26: Sending DeleteTablet for 3 replicas of tablet 5dc80ae8e16943d9b2f5bd1cb4c9f09a
I20260810 07:25:38.704047 25742 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dc80ae8e16943d9b2f5bd1cb4c9f09a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:38 UTC) from {username='oryx', principal='oryx/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:41500
I20260810 07:25:38.704087 25957 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dc80ae8e16943d9b2f5bd1cb4c9f09a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:38 UTC) from {username='oryx', principal='oryx/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:38940
I20260810 07:25:38.704285 26429 tablet_replica.cc:330] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193: stopping tablet replica
I20260810 07:25:38.704281 26431 tablet_replica.cc:330] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 1f5b3dd52a30437bb9dc4217ecee6741: stopping tablet replica
I20260810 07:25:38.704411 26429 raft_consensus.cc:2229] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:25:38.704439 26431 raft_consensus.cc:2229] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 1f5b3dd52a30437bb9dc4217ecee6741 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:25:38.704553 26431 raft_consensus.cc:2258] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 1f5b3dd52a30437bb9dc4217ecee6741 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:38.704738 26429 raft_consensus.cc:2258] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:38.705587 26429 ts_tablet_manager.cc:1905] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:38.705747 26431 ts_tablet_manager.cc:1905] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 1f5b3dd52a30437bb9dc4217ecee6741: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:38.707852 26431 ts_tablet_manager.cc:1918] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 1f5b3dd52a30437bb9dc4217ecee6741: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 07:25:38.707952 26431 log.cc:1196] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 1f5b3dd52a30437bb9dc4217ecee6741: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-0/wal/wals/5dc80ae8e16943d9b2f5bd1cb4c9f09a
I20260810 07:25:38.708195 26431 ts_tablet_manager.cc:1939] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 1f5b3dd52a30437bb9dc4217ecee6741: Deleting consensus metadata
I20260810 07:25:38.708253 26429 ts_tablet_manager.cc:1918] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 07:25:38.708343 26429 log.cc:1196] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-1/wal/wals/5dc80ae8e16943d9b2f5bd1cb4c9f09a
I20260810 07:25:38.708618 26429 ts_tablet_manager.cc:1939] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 03e839a9c2594f89b0e4f6cddff8a193: Deleting consensus metadata
I20260810 07:25:38.708981 26177 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dc80ae8e16943d9b2f5bd1cb4c9f09a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:25:38 UTC) from {username='oryx', principal='oryx/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:58434
I20260810 07:25:38.709033 25496 catalog_manager.cc:4841] TS 1f5b3dd52a30437bb9dc4217ecee6741 (127.12.157.193:42927): tablet 5dc80ae8e16943d9b2f5bd1cb4c9f09a (table test-table [id=2ba512ec64a449bd97acaab5fc2c6e89]) successfully deleted
I20260810 07:25:38.709193 26432 tablet_replica.cc:330] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 96142be645164fd7887695f0986f8965: stopping tablet replica
I20260810 07:25:38.709343 26432 raft_consensus.cc:2229] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 96142be645164fd7887695f0986f8965 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:25:38.709476 26432 raft_consensus.cc:2258] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 96142be645164fd7887695f0986f8965 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:25:38.709479 25495 catalog_manager.cc:4841] TS 03e839a9c2594f89b0e4f6cddff8a193 (127.12.157.194:38095): tablet 5dc80ae8e16943d9b2f5bd1cb4c9f09a (table test-table [id=2ba512ec64a449bd97acaab5fc2c6e89]) successfully deleted
I20260810 07:25:38.709975 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 25639
I20260810 07:25:38.710327 26432 ts_tablet_manager.cc:1905] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 96142be645164fd7887695f0986f8965: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:25:38.722981 26432 ts_tablet_manager.cc:1918] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 96142be645164fd7887695f0986f8965: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 07:25:38.723126 26432 log.cc:1196] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 96142be645164fd7887695f0986f8965: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786346694975122-12919-0/minicluster-data/ts-2/wal/wals/5dc80ae8e16943d9b2f5bd1cb4c9f09a
I20260810 07:25:38.723519 26432 ts_tablet_manager.cc:1939] T 5dc80ae8e16943d9b2f5bd1cb4c9f09a P 96142be645164fd7887695f0986f8965: Deleting consensus metadata
I20260810 07:25:38.724506 25500 catalog_manager.cc:4841] TS 96142be645164fd7887695f0986f8965 (127.12.157.195:41691): tablet 5dc80ae8e16943d9b2f5bd1cb4c9f09a (table test-table [id=2ba512ec64a449bd97acaab5fc2c6e89]) successfully deleted
I20260810 07:25:38.734056 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 25855
I20260810 07:25:38.750504 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 26072
I20260810 07:25:38.767544 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 25470
2026-08-10T07:25:38Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3755 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:25:38 dist-test-slave-ffrd krb5kdc[26497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:25:38 dist-test-slave-ffrd krb5kdc[26497](info): set up 2 sockets
Aug 10 07:25:38 dist-test-slave-ffrd krb5kdc[26497](info): commencing operation
krb5kdc: starting...
W20260810 07:25:40.067384 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.271s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:25:40 dist-test-slave-ffrd krb5kdc[26497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:25:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:25:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:25:40.204344 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:40143
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:44777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:40143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:40.321673 26514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:40.322000 26514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:40.322089 26514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:40.326215 26514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:25:40.326301 26514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:40.326339 26514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:40.326402 26514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:25:40.326442 26514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:25:40.331691 26514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:44777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:40143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:40143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:40.332787 26514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:40.333885 26514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:40.342276 26527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:40.342306 26521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:40.342289 26514 server_base.cc:1034] running on GCE node
W20260810 07:25:40.342298 26524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:40.343076 26514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:40.344118 26514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:40.345314 26514 hybrid_clock.cc:648] HybridClock initialized: now 1786346740345296 us; error 30 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T07:25:40Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1579 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:25:40 dist-test-slave-ffrd krb5kdc[26535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:25:40 dist-test-slave-ffrd krb5kdc[26535](info): set up 2 sockets
Aug 10 07:25:40 dist-test-slave-ffrd krb5kdc[26535](info): commencing operation
krb5kdc: starting...
W20260810 07:25:41.617293 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.241s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:25:41 dist-test-slave-ffrd krb5kdc[26535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:25:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:25:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:25:41.749855 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:37163
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:42423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:37163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:41.863462 26556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:41.863762 26556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:41.863829 26556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:41.867795 26556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:25:41.867893 26556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:41.867947 26556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:41.867981 26556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:25:41.868012 26556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:25:41.873198 26556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:42423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:37163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:37163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:41.874282 26556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:41.875480 26556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:41.884271 26565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:41.884240 26561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:41.884392 26562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:41.884503 26556 server_base.cc:1034] running on GCE node
I20260810 07:25:41.885180 26556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:41.886369 26556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:41.887591 26556 hybrid_clock.cc:648] HybridClock initialized: now 1786346741887578 us; error 30 us; skew 500 ppm
Aug 10 07:25:41 dist-test-slave-ffrd krb5kdc[26535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346741, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:41.894068 26556 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:25:41.895038 26556 webserver.cc:460] Webserver started at http://127.12.157.254:36165/ using document root <none> and password file <none>
I20260810 07:25:41.895737 26556 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:41.895854 26556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:41.896113 26556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:41.898367 26556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "36db11bc05454d638084cc64daa2110e"
format_stamp: "Formatted at 2026-08-10 07:25:41 on dist-test-slave-ffrd"
server_key: "0347fb3a06db43cdb6210353eff9e2f7"
server_key_iv: "b54051e21b7e09e7a0f3e512b280b201"
server_key_version: "encryptionkey@0"
I20260810 07:25:41.898978 26556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "36db11bc05454d638084cc64daa2110e"
format_stamp: "Formatted at 2026-08-10 07:25:41 on dist-test-slave-ffrd"
server_key: "0347fb3a06db43cdb6210353eff9e2f7"
server_key_iv: "b54051e21b7e09e7a0f3e512b280b201"
server_key_version: "encryptionkey@0"
I20260810 07:25:41.902869 26556 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 07:25:41.905417 26582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:41.906714 26556 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:25:41.906858 26556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "36db11bc05454d638084cc64daa2110e"
format_stamp: "Formatted at 2026-08-10 07:25:41 on dist-test-slave-ffrd"
server_key: "0347fb3a06db43cdb6210353eff9e2f7"
server_key_iv: "b54051e21b7e09e7a0f3e512b280b201"
server_key_version: "encryptionkey@0"
I20260810 07:25:41.906996 26556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:41.933468 26556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:41.936635 26556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:41.936859 26556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:41.944108 26556 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:37163
I20260810 07:25:41.944124 26658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:37163 every 8 connection(s)
I20260810 07:25:41.945127 26556 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:25:41.947618 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 26556
I20260810 07:25:41.947724 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:25:41.948006 26659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:41.948042 12919 external_mini_cluster.cc:1428] Setting key 296dd1102cf169e79c0b2979c5d3c8dd
I20260810 07:25:41.955148 26659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e: Bootstrap starting.
I20260810 07:25:41.957828 26659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e: Neither blocks nor log segments found. Creating new log.
Aug 10 07:25:41 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:41.958776 26659 log.cc:824] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:41.961169 26659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e: No bootstrap required, opened a new log
I20260810 07:25:41.963747 26663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:41.949457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52888 (local address 127.12.157.254:37163)
0810 07:25:41.949854 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:41.949877 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:41.949912 (+ 35us) server_negotiation.cc:407] Connection header received
0810 07:25:41.950773 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:41.950793 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:41.951115 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:41.951401 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:41.952443 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:41.953578 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:41.954764 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:41.955233 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:41.955432 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:41.958572 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:41.958577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:41.958591 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:41.958619 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:41.960448 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:41.961057 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:41.961062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:41.961067 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:41.961183 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:41.961579 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:41.961582 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:41.961583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:41.962006 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:41.962156 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:41.962492 (+ 336us) server_negotiation.cc:299] Negotiation successful
0810 07:25:41.963008 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":133,"threads_started":1}
I20260810 07:25:41.965534 26659 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36db11bc05454d638084cc64daa2110e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37163 } }
I20260810 07:25:41.965833 26659 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:41.966002 26659 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36db11bc05454d638084cc64daa2110e, State: Initialized, Role: FOLLOWER
I20260810 07:25:41.966466 26659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36db11bc05454d638084cc64daa2110e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37163 } }
I20260810 07:25:41.966605 26659 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:25:41.966657 26659 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:25:41.966733 26659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:41.967747 26659 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36db11bc05454d638084cc64daa2110e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37163 } }
I20260810 07:25:41.968077 26659 leader_election.cc:302] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36db11bc05454d638084cc64daa2110e; no voters:
I20260810 07:25:41.968386 26659 leader_election.cc:288] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:25:41.968552 26666 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:41.968819 26666 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [term 1 LEADER]: Becoming Leader. State: Replica: 36db11bc05454d638084cc64daa2110e, State: Running, Role: LEADER
I20260810 07:25:41.969206 26666 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36db11bc05454d638084cc64daa2110e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37163 } }
I20260810 07:25:41.969837 26659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:25:41.970917 26667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36db11bc05454d638084cc64daa2110e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36db11bc05454d638084cc64daa2110e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37163 } } }
I20260810 07:25:41.971131 26667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:41.971510 26668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36db11bc05454d638084cc64daa2110e. Latest consensus state: current_term: 1 leader_uuid: "36db11bc05454d638084cc64daa2110e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36db11bc05454d638084cc64daa2110e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37163 } } }
I20260810 07:25:41.971674 26668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:41.971662 26683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:25:41.975625 26683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:25:41.982023 26683 catalog_manager.cc:1347] Generated new cluster ID: c9e2d28a315f4b6489cb84eadb7f49a8
I20260810 07:25:41.982106 26683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:25:42.022795 26683 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:25:42.024173 26683 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:25:42.034567 26683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e: Generated new TSK 0
I20260810 07:25:42.035480 26683 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
I20260810 07:25:42.080289 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:0
--local_ip_for_outbound_sockets=127.12.157.193
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37163
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:42423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:42.195554 26700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:42.195807 26700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:42.195850 26700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:42.199983 26700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:42.200050 26700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:42.200146 26700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:25:42.205335 26700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:42423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:42.206393 26700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:42.207469 26700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:42.217182 26708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:42.217195 26714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:42.217242 26700 server_base.cc:1034] running on GCE node
W20260810 07:25:42.217195 26710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:42.217821 26700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:42.218451 26700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:42.219647 26700 hybrid_clock.cc:648] HybridClock initialized: now 1786346742219632 us; error 26 us; skew 500 ppm
Aug 10 07:25:42 dist-test-slave-ffrd krb5kdc[26535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346742, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:42.222409 26700 init.cc:377] Logged in from keytab as kudu/127.12.157.193@KRBTEST.COM (short username kudu)
I20260810 07:25:42.223263 26700 webserver.cc:460] Webserver started at http://127.12.157.193:40243/ using document root <none> and password file <none>
I20260810 07:25:42.223820 26700 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:42.223917 26700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:42.224141 26700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:42.226193 26700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/data/instance:
uuid: "1eb897a898c443a79f1ec03072610ace"
format_stamp: "Formatted at 2026-08-10 07:25:42 on dist-test-slave-ffrd"
server_key: "931bd5e10d6cb58775493de1e42871c8"
server_key_iv: "c4a83227884c5487053702e4165980b2"
server_key_version: "encryptionkey@0"
I20260810 07:25:42.226768 26700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance:
uuid: "1eb897a898c443a79f1ec03072610ace"
format_stamp: "Formatted at 2026-08-10 07:25:42 on dist-test-slave-ffrd"
server_key: "931bd5e10d6cb58775493de1e42871c8"
server_key_iv: "c4a83227884c5487053702e4165980b2"
server_key_version: "encryptionkey@0"
I20260810 07:25:42.230513 26700 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 07:25:42.233043 26723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:42.234198 26700 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:25:42.234352 26700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "1eb897a898c443a79f1ec03072610ace"
format_stamp: "Formatted at 2026-08-10 07:25:42 on dist-test-slave-ffrd"
server_key: "931bd5e10d6cb58775493de1e42871c8"
server_key_iv: "c4a83227884c5487053702e4165980b2"
server_key_version: "encryptionkey@0"
I20260810 07:25:42.234475 26700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:42.251398 26700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:42.254673 26700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:42.254911 26700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:42.255577 26700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:42.256512 26700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:42.256594 26700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:42.256670 26700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:42.256724 26700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:42.267098 26700 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:39135
I20260810 07:25:42.267122 26897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:39135 every 8 connection(s)
I20260810 07:25:42.268165 26700 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
Aug 10 07:25:42 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346742, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:42.278151 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 26700
I20260810 07:25:42.278301 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance
I20260810 07:25:42.278627 12919 external_mini_cluster.cc:1428] Setting key b931ffcb27469fad5f6317cbce025be2
I20260810 07:25:42.282591 26663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:42.270202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:34317 (local address 127.12.157.254:37163)
0810 07:25:42.270359 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:42.270364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:42.271374 (+ 1010us) server_negotiation.cc:407] Connection header received
0810 07:25:42.272765 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:42.272768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:42.272816 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:42.272901 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:42.274157 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:42.274722 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:42.275830 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:42.276109 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:42.276194 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:42.278850 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:42.278854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:42.278857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:42.278889 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:42.280413 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:42.280981 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:42.280988 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:42.280990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:42.281075 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:42.281456 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:42.281460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:42.281462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:42.281655 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:42.281772 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:42.282288 (+ 516us) server_negotiation.cc:299] Negotiation successful
0810 07:25:42.282423 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 07:25:42.283479 26903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:42.270696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.193:34317)
0810 07:25:42.271207 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:42.271271 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:42.272545 (+ 1274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:42.273129 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:42.273136 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:42.273512 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:42.274002 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:42.274016 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:42.274834 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:42.274836 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:42.275714 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:42.275724 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:42.276246 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:42.276249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:42.276387 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:42.277087 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:42.277104 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:42.278648 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:42.280546 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:42.280552 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:42.280569 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:42.280835 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:42.281177 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:42.281181 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:42.281183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:42.281333 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:42.281771 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:42.281778 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:42.282042 (+ 264us) client_negotiation.cc:769] Sending connection context
0810 07:25:42.282245 (+ 203us) client_negotiation.cc:240] Negotiation successful
0810 07:25:42.282604 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":121,"threads_started":1}
I20260810 07:25:42.284878 26901 heartbeater.cc:347] Connected to a master server at 127.12.157.254:37163
I20260810 07:25:42.285164 26901 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:42.285688 26901 heartbeater.cc:511] Master 127.12.157.254:37163 requested a full tablet report, sending...
I20260810 07:25:42.287364 26604 ts_manager.cc:194] Registered new tserver with Master: 1eb897a898c443a79f1ec03072610ace (127.12.157.193:39135)
I20260810 07:25:42.288509 26604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.193@KRBTEST.COM'} at 127.12.157.193:34317
WARNING: no policy specified for kudu/127.12.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
I20260810 07:25:42.308660 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:0
--local_ip_for_outbound_sockets=127.12.157.194
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37163
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:42423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:42.429697 26908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:42.429988 26908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:42.430035 26908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:42.434379 26908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:42.434456 26908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:42.434518 26908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:25:42.440310 26908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:42423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:42.441537 26908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:42.442608 26908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:42.452571 26915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:42.452581 26914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:42.452898 26908 server_base.cc:1034] running on GCE node
W20260810 07:25:42.452590 26921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:42.453442 26908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:42.454106 26908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:42.455307 26908 hybrid_clock.cc:648] HybridClock initialized: now 1786346742455289 us; error 28 us; skew 500 ppm
Aug 10 07:25:42 dist-test-slave-ffrd krb5kdc[26535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346742, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:42.458526 26908 init.cc:377] Logged in from keytab as kudu/127.12.157.194@KRBTEST.COM (short username kudu)
I20260810 07:25:42.459510 26908 webserver.cc:460] Webserver started at http://127.12.157.194:35099/ using document root <none> and password file <none>
I20260810 07:25:42.460120 26908 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:42.460219 26908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:42.460480 26908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:42.462651 26908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/data/instance:
uuid: "1feb5fa5fbd6431c8bdbd18269f8905e"
format_stamp: "Formatted at 2026-08-10 07:25:42 on dist-test-slave-ffrd"
server_key: "b8ba88a7888fabf9daba1d8cf5b45d33"
server_key_iv: "0ae10f27ee6c5cfb645c910036b06aca"
server_key_version: "encryptionkey@0"
I20260810 07:25:42.463631 26908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance:
uuid: "1feb5fa5fbd6431c8bdbd18269f8905e"
format_stamp: "Formatted at 2026-08-10 07:25:42 on dist-test-slave-ffrd"
server_key: "b8ba88a7888fabf9daba1d8cf5b45d33"
server_key_iv: "0ae10f27ee6c5cfb645c910036b06aca"
server_key_version: "encryptionkey@0"
I20260810 07:25:42.467897 26908 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260810 07:25:42.470897 26929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:42.472399 26908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 07:25:42.472505 26908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "1feb5fa5fbd6431c8bdbd18269f8905e"
format_stamp: "Formatted at 2026-08-10 07:25:42 on dist-test-slave-ffrd"
server_key: "b8ba88a7888fabf9daba1d8cf5b45d33"
server_key_iv: "0ae10f27ee6c5cfb645c910036b06aca"
server_key_version: "encryptionkey@0"
I20260810 07:25:42.472666 26908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:42.498502 26908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:42.501820 26908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:42.502039 26908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:42.502698 26908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:42.503571 26908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:42.503619 26908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:42.503662 26908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:42.503723 26908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:42.513576 26908 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:36053
I20260810 07:25:42.513612 27117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:36053 every 8 connection(s)
I20260810 07:25:42.514449 26908 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:25:42.516712 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 26908
I20260810 07:25:42.516868 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance
I20260810 07:25:42.517187 12919 external_mini_cluster.cc:1428] Setting key 9290a28da2a581d3f09037a6df9e7719
Aug 10 07:25:42 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346742, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.157.195@KRBTEST.COM; defaulting to no policy
I20260810 07:25:42.530717 26663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:42.516296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:48077 (local address 127.12.157.254:37163)
0810 07:25:42.516464 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:42.516468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:42.517958 (+ 1490us) server_negotiation.cc:407] Connection header received
0810 07:25:42.518819 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:42.518823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:42.518876 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:42.518970 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:42.520228 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:42.521099 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:42.522881 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:42.523104 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:42.523187 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:42.526491 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:42.526498 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:42.526505 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:42.526541 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:42.528225 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:42.528835 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:42.528839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:42.528842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:42.528959 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:42.529516 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:42.529521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:42.529523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:42.529801 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:42.529890 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:42.530470 (+ 580us) server_negotiation.cc:299] Negotiation successful
0810 07:25:42.530587 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 07:25:42.531669 27120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:42.516673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.194:48077)
0810 07:25:42.517744 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:42.517798 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:42.518619 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:42.519174 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:42.519182 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:42.519565 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:42.520068 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:42.520083 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:42.521234 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:42.521240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:42.522743 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:42.522759 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:42.523270 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:42.523277 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:42.523530 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:42.524191 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:42.524219 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:42.526291 (+ 2072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:42.528360 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:42.528378 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:42.528394 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:42.528715 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:42.529095 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:42.529098 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:42.529099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:42.529249 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:42.529918 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:42.529923 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:42.530168 (+ 245us) client_negotiation.cc:769] Sending connection context
0810 07:25:42.530345 (+ 177us) client_negotiation.cc:240] Negotiation successful
0810 07:25:42.530698 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"thread_start_us":697,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.195@KRBTEST.COM" created.
I20260810 07:25:42.533067 27118 heartbeater.cc:347] Connected to a master server at 127.12.157.254:37163
I20260810 07:25:42.533319 27118 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:42.533912 27118 heartbeater.cc:511] Master 127.12.157.254:37163 requested a full tablet report, sending...
I20260810 07:25:42.535038 26604 ts_manager.cc:194] Registered new tserver with Master: 1feb5fa5fbd6431c8bdbd18269f8905e (127.12.157.194:36053)
I20260810 07:25:42.535625 26604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.194@KRBTEST.COM'} at 127.12.157.194:48077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
I20260810 07:25:42.548237 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:0
--local_ip_for_outbound_sockets=127.12.157.195
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37163
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:42423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:42.664671 27127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:42.665002 27127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:42.665097 27127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:42.669191 27127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:42.669251 27127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:42.669351 27127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:25:42.674991 27127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:42423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:42.676141 27127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:42.677353 27127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:42.687186 27135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:42.687301 27134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:42.687171 27137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:42.687433 27127 server_base.cc:1034] running on GCE node
I20260810 07:25:42.687916 27127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:42.688561 27127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:42.689730 27127 hybrid_clock.cc:648] HybridClock initialized: now 1786346742689712 us; error 27 us; skew 500 ppm
Aug 10 07:25:42 dist-test-slave-ffrd krb5kdc[26535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346742, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:42.692579 27127 init.cc:377] Logged in from keytab as kudu/127.12.157.195@KRBTEST.COM (short username kudu)
I20260810 07:25:42.693523 27127 webserver.cc:460] Webserver started at http://127.12.157.195:42899/ using document root <none> and password file <none>
I20260810 07:25:42.694090 27127 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:42.694152 27127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:42.694406 27127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:42.696489 27127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/data/instance:
uuid: "e9f7a12ec4164a339a72ac228cfb4a32"
format_stamp: "Formatted at 2026-08-10 07:25:42 on dist-test-slave-ffrd"
server_key: "9be803fb953a2e3453f23ad3b4c126fc"
server_key_iv: "35e91c09b5e2435d8db29a248ffc3bc7"
server_key_version: "encryptionkey@0"
I20260810 07:25:42.697194 27127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance:
uuid: "e9f7a12ec4164a339a72ac228cfb4a32"
format_stamp: "Formatted at 2026-08-10 07:25:42 on dist-test-slave-ffrd"
server_key: "9be803fb953a2e3453f23ad3b4c126fc"
server_key_iv: "35e91c09b5e2435d8db29a248ffc3bc7"
server_key_version: "encryptionkey@0"
I20260810 07:25:42.701151 27127 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 07:25:42.703819 27146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:42.705327 27127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:25:42.705476 27127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "e9f7a12ec4164a339a72ac228cfb4a32"
format_stamp: "Formatted at 2026-08-10 07:25:42 on dist-test-slave-ffrd"
server_key: "9be803fb953a2e3453f23ad3b4c126fc"
server_key_iv: "35e91c09b5e2435d8db29a248ffc3bc7"
server_key_version: "encryptionkey@0"
I20260810 07:25:42.705608 27127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:42.729609 27127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:42.733144 27127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:42.733355 27127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:42.733994 27127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:42.734858 27127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:42.734931 27127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:42.735001 27127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:42.735047 27127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:42.744797 27127 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:37699
I20260810 07:25:42.744817 27307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:37699 every 8 connection(s)
I20260810 07:25:42.745764 27127 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
Aug 10 07:25:42 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346742, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:25:42.756138 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 27127
I20260810 07:25:42.756352 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance
I20260810 07:25:42.756783 12919 external_mini_cluster.cc:1428] Setting key b1c229d1bf10041e79d810f99eeb0cd6
I20260810 07:25:42.760630 26663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:42.747685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:53585 (local address 127.12.157.254:37163)
0810 07:25:42.747895 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:42.747900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:42.748725 (+ 825us) server_negotiation.cc:407] Connection header received
0810 07:25:42.750051 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:42.750054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:42.750104 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:42.750179 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:42.751419 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:42.751975 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:42.753225 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:42.753581 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:42.753675 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:42.756744 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:42.756747 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:42.756749 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:42.756775 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:42.758552 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:42.759120 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:42.759124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:42.759125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:42.759220 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:42.759598 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:42.759602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:42.759603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:42.759828 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:42.759934 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:42.760407 (+ 473us) server_negotiation.cc:299] Negotiation successful
0810 07:25:42.760514 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 07:25:42.761591 27310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:42.748085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.195:53585)
0810 07:25:42.748565 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:42.748607 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:42.749846 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:42.750362 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:42.750369 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:42.750729 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:42.751245 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:42.751259 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:42.752113 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:42.752117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:42.753104 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:42.753115 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:42.753759 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:42.753762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:42.753945 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:42.754643 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:42.754660 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:42.756544 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:42.758675 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:42.758680 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:42.758693 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:42.759011 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:42.759328 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:25:42.759331 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:25:42.759332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:25:42.759484 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:25:42.759937 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:42.759943 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:42.760186 (+ 243us) client_negotiation.cc:769] Sending connection context
0810 07:25:42.760349 (+ 163us) client_negotiation.cc:240] Negotiation successful
0810 07:25:42.760628 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":123,"threads_started":1}
I20260810 07:25:42.762733 27308 heartbeater.cc:347] Connected to a master server at 127.12.157.254:37163
I20260810 07:25:42.762987 27308 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:42.763499 27308 heartbeater.cc:511] Master 127.12.157.254:37163 requested a full tablet report, sending...
I20260810 07:25:42.764595 26604 ts_manager.cc:194] Registered new tserver with Master: e9f7a12ec4164a339a72ac228cfb4a32 (127.12.157.195:37699)
I20260810 07:25:42.765203 26604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.195@KRBTEST.COM'} at 127.12.157.195:53585
I20260810 07:25:42.770964 12919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.193.keytab.
Entry for principal oryx/127.12.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.193.keytab.
WARNING: no policy specified for oryx/127.12.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.194.keytab.
Entry for principal oryx/127.12.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.194.keytab.
WARNING: no policy specified for oryx/127.12.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.195.keytab.
Entry for principal oryx/127.12.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.195.keytab.
I20260810 07:25:42.856313 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 26700
I20260810 07:25:42.869112 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 26908
I20260810 07:25:42.888671 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 27127
I20260810 07:25:42.897356 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 26556
I20260810 07:25:42.907612 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:37163
--webserver_interface=127.12.157.254
--webserver_port=36165
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:42423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:37163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:25:43.024127 27318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:43.024499 27318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:43.024585 27318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:43.028694 27318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:25:43.028805 27318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:43.028846 27318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:43.028879 27318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:25:43.028918 27318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:25:43.034708 27318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:42423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:37163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:37163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=36165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:43.035887 27318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:43.037032 27318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:43.046172 27327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:43.046164 27331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:43.046239 27318 server_base.cc:1034] running on GCE node
W20260810 07:25:43.046167 27328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:43.047046 27318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:43.048162 27318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:43.049353 27318 hybrid_clock.cc:648] HybridClock initialized: now 1786346743049344 us; error 32 us; skew 500 ppm
Aug 10 07:25:43 dist-test-slave-ffrd krb5kdc[26535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346743, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:43.052171 27318 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:25:43.053190 27318 webserver.cc:460] Webserver started at http://127.12.157.254:36165/ using document root <none> and password file <none>
I20260810 07:25:43.053763 27318 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:43.053889 27318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:43.057765 27318 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260810 07:25:43.059957 27349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:43.061400 27318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 07:25:43.061537 27318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "36db11bc05454d638084cc64daa2110e"
format_stamp: "Formatted at 2026-08-10 07:25:41 on dist-test-slave-ffrd"
server_key: "0347fb3a06db43cdb6210353eff9e2f7"
server_key_iv: "b54051e21b7e09e7a0f3e512b280b201"
server_key_version: "encryptionkey@0"
I20260810 07:25:43.062104 27318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:43.080905 27318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:43.084437 27318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:43.084662 27318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:43.092314 27318 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:37163
I20260810 07:25:43.092326 27424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:37163 every 8 connection(s)
I20260810 07:25:43.093500 27318 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:25:43.094691 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 27318
I20260810 07:25:43.095171 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:39135
--local_ip_for_outbound_sockets=127.12.157.193
--tserver_master_addrs=127.12.157.254:37163
--webserver_port=40243
--webserver_interface=127.12.157.193
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:42423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.157.193
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.193.keytab with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:25:43.098078 27425 sys_catalog.cc:263] Verifying existing consensus state
I20260810 07:25:43.099771 27425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e: Bootstrap starting.
I20260810 07:25:43.119357 27425 log.cc:824] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:43.126662 27425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 07:25:43.127074 27425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e: Bootstrap complete.
I20260810 07:25:43.131577 27425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36db11bc05454d638084cc64daa2110e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37163 } }
I20260810 07:25:43.132017 27425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36db11bc05454d638084cc64daa2110e, State: Initialized, Role: FOLLOWER
I20260810 07:25:43.132515 27425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36db11bc05454d638084cc64daa2110e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37163 } }
I20260810 07:25:43.132665 27425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:25:43.132772 27425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:25:43.132901 27425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [term 1 FOLLOWER]: Advancing to term 2
I20260810 07:25:43.134902 27425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36db11bc05454d638084cc64daa2110e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37163 } }
I20260810 07:25:43.135250 27425 leader_election.cc:302] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36db11bc05454d638084cc64daa2110e; no voters:
I20260810 07:25:43.135609 27425 leader_election.cc:288] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 07:25:43.135756 27434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [term 2 FOLLOWER]: Leader election won for term 2
I20260810 07:25:43.136008 27434 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [term 2 LEADER]: Becoming Leader. State: Replica: 36db11bc05454d638084cc64daa2110e, State: Running, Role: LEADER
I20260810 07:25:43.136404 27434 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36db11bc05454d638084cc64daa2110e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37163 } }
I20260810 07:25:43.136870 27425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:25:43.137996 27435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "36db11bc05454d638084cc64daa2110e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36db11bc05454d638084cc64daa2110e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37163 } } }
I20260810 07:25:43.138124 27435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:43.138518 27437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36db11bc05454d638084cc64daa2110e. Latest consensus state: current_term: 2 leader_uuid: "36db11bc05454d638084cc64daa2110e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36db11bc05454d638084cc64daa2110e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37163 } } }
I20260810 07:25:43.138692 27437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:43.138971 27450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:25:43.142295 27450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:25:43.143081 27450 catalog_manager.cc:1259] Loaded cluster ID: c9e2d28a315f4b6489cb84eadb7f49a8
I20260810 07:25:43.143154 27450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:25:43.145742 27450 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:25:43.146407 27450 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 36db11bc05454d638084cc64daa2110e: Loaded TSK: 0
I20260810 07:25:43.147611 27450 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 07:25:43.220578 27430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:43.220830 27430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:43.220873 27430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:43.224825 27430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:43.224881 27430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:43.224992 27430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:25:43.230161 27430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:42423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.193.keytab
--principal=oryx/127.12.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:39135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=40243
--tserver_master_addrs=127.12.157.254:37163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:43.231124 27430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:43.232110 27430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:43.242488 27478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:43.242491 27476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:43.242854 27475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:43.243453 27430 server_base.cc:1034] running on GCE node
I20260810 07:25:43.243839 27430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:43.244524 27430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:43.245705 27430 hybrid_clock.cc:648] HybridClock initialized: now 1786346743245675 us; error 45 us; skew 500 ppm
Aug 10 07:25:43 dist-test-slave-ffrd krb5kdc[26535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346743, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:43.248526 27430 init.cc:377] Logged in from keytab as oryx/127.12.157.193@KRBTEST.COM (short username oryx)
I20260810 07:25:43.249549 27430 webserver.cc:460] Webserver started at http://127.12.157.193:40243/ using document root <none> and password file <none>
I20260810 07:25:43.250108 27430 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:43.250183 27430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:43.254236 27430 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 07:25:43.256326 27493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:43.257627 27430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:25:43.257722 27430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "1eb897a898c443a79f1ec03072610ace"
format_stamp: "Formatted at 2026-08-10 07:25:42 on dist-test-slave-ffrd"
server_key: "931bd5e10d6cb58775493de1e42871c8"
server_key_iv: "c4a83227884c5487053702e4165980b2"
server_key_version: "encryptionkey@0"
I20260810 07:25:43.258278 27430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:43.272569 27430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:43.275631 27430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:43.275826 27430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:43.276484 27430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:43.277429 27430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:43.277478 27430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:43.277519 27430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:43.277557 27430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:43.287480 27430 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:39135
I20260810 07:25:43.287521 27665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:39135 every 8 connection(s)
I20260810 07:25:43.288481 27430 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:25:43.293107 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 27430
I20260810 07:25:43.293560 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:36053
--local_ip_for_outbound_sockets=127.12.157.194
--tserver_master_addrs=127.12.157.254:37163
--webserver_port=35099
--webserver_interface=127.12.157.194
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:42423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.157.194
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.194.keytab with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 07:25:43 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:43 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 07:25:43.302285 27670 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:43.290696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:60821 (local address 127.12.157.254:37163)
0810 07:25:43.291250 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:43.291293 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:43.291498 (+ 205us) server_negotiation.cc:407] Connection header received
0810 07:25:43.293008 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:43.293022 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:43.293358 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:43.293670 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:43.294800 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.295746 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.296881 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.297406 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.297585 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:43.301609 (+ 4024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.193:60821: BlockingRecv error: recv got EOF from 127.12.157.193:60821 (error 108)
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":141,"threads_started":1}
I20260810 07:25:43.302435 27671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:43.290793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.193:60821)
0810 07:25:43.291301 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:43.291376 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:43.292551 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:43.293741 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:43.293748 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:43.294180 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:43.294629 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:43.294642 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.295898 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.295904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:43.296741 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:43.296750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.297572 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.297576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:43.297765 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:43.298512 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:43.298529 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:43.301400 (+ 2871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":146,"threads_started":1}
W20260810 07:25:43.303023 27666 heartbeater.cc:644] Failed to heartbeat to 127.12.157.254:37163 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.157.254:37163: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Aug 10 07:25:43 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:43 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:43.308651 27671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:43.303491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.193:51011)
0810 07:25:43.303615 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:43.303629 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:43.303709 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:43.303959 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:43.303962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:43.304124 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:43.304226 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:43.304231 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.304981 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.304985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:43.305520 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:43.305528 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.305991 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.305995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:43.306086 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:43.306595 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:43.306607 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:43.308504 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 07:25:43.308734 27670 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:43.303595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:51011 (local address 127.12.157.254:37163)
0810 07:25:43.303710 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:43.303714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:43.303730 (+ 16us) server_negotiation.cc:407] Connection header received
0810 07:25:43.303794 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:43.303797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:43.303848 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:43.303945 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:43.304341 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.304842 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.305644 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.305874 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.305991 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:43.308621 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.193:51011: BlockingRecv error: recv got EOF from 127.12.157.193:51011 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 10 07:25:43 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:43 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:43.314512 27671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:43.309196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.193:37181)
0810 07:25:43.309327 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:43.309339 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:43.309398 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:43.309607 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:43.309609 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:43.309737 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:43.309805 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:43.309810 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.310793 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.310796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:43.311294 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:43.311302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.312035 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.312038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:43.312093 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:43.312589 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:43.312601 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:43.314426 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 07:25:43.314707 27670 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:43.309271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:37181 (local address 127.12.157.254:37163)
0810 07:25:43.309372 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:43.309376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:43.309390 (+ 14us) server_negotiation.cc:407] Connection header received
0810 07:25:43.309462 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:43.309464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:43.309504 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:43.309590 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:43.309908 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.310678 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.311517 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.311911 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.312057 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:43.314600 (+ 2543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.193:37181: BlockingRecv error: recv got EOF from 127.12.157.193:37181 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 07:25:43.314716 27666 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 07:25:43.413460 27672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:43.413830 27672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:43.413915 27672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:43.418139 27672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:43.418253 27672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:43.418354 27672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:25:43.423851 27672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:42423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.194.keytab
--principal=oryx/127.12.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:36053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=35099
--tserver_master_addrs=127.12.157.254:37163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:43.425108 27672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:43.426213 27672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 07:25:43.436522 27672 server_base.cc:1034] running on GCE node
W20260810 07:25:43.436467 27686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:43.436479 27689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:43.436487 27682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:43.437305 27672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:43.438150 27672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:43.439324 27672 hybrid_clock.cc:648] HybridClock initialized: now 1786346743439294 us; error 57 us; skew 500 ppm
Aug 10 07:25:43 dist-test-slave-ffrd krb5kdc[26535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346743, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:43.442631 27672 init.cc:377] Logged in from keytab as oryx/127.12.157.194@KRBTEST.COM (short username oryx)
I20260810 07:25:43.443652 27672 webserver.cc:460] Webserver started at http://127.12.157.194:35099/ using document root <none> and password file <none>
I20260810 07:25:43.444270 27672 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:43.444386 27672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:43.448491 27672 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260810 07:25:43.450839 27697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:43.452152 27672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 07:25:43.452309 27672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "1feb5fa5fbd6431c8bdbd18269f8905e"
format_stamp: "Formatted at 2026-08-10 07:25:42 on dist-test-slave-ffrd"
server_key: "b8ba88a7888fabf9daba1d8cf5b45d33"
server_key_iv: "0ae10f27ee6c5cfb645c910036b06aca"
server_key_version: "encryptionkey@0"
I20260810 07:25:43.452917 27672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:43.471968 27672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:43.475435 27672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:43.475653 27672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:43.476295 27672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:43.477203 27672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:43.477274 27672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:43.477348 27672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:43.477392 27672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:43.489212 27672 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:36053
I20260810 07:25:43.489252 27880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:36053 every 8 connection(s)
I20260810 07:25:43.490414 27672 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:25:43.491467 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 27672
I20260810 07:25:43.492028 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:37699
--local_ip_for_outbound_sockets=127.12.157.195
--tserver_master_addrs=127.12.157.254:37163
--webserver_port=42899
--webserver_interface=127.12.157.195
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:42423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.157.195
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.195.keytab with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 07:25:43 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:43 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 07:25:43.505618 27670 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:43.494819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:39039 (local address 127.12.157.254:37163)
0810 07:25:43.494980 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:43.494984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:43.496354 (+ 1370us) server_negotiation.cc:407] Connection header received
0810 07:25:43.497393 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:43.497397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:43.497465 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:43.497550 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:43.498898 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.499444 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.500654 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.500884 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.501062 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:43.505483 (+ 4421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.194:39039: BlockingRecv error: recv got EOF from 127.12.157.194:39039 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 07:25:43.506318 27884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:43.495362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.194:39039)
0810 07:25:43.496184 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:43.496227 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:43.497180 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:43.497733 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:43.497740 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:43.498204 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:43.498736 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:43.498750 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.499588 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.499597 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:43.500512 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:43.500521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.501115 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.501120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:43.501303 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:43.502131 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:43.502149 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:43.505349 (+ 3200us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":171,"threads_started":1}
W20260810 07:25:43.506853 27881 heartbeater.cc:644] Failed to heartbeat to 127.12.157.254:37163 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.157.254:37163: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Aug 10 07:25:43 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:43 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:43.513206 27884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:43.507363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.194:34409)
0810 07:25:43.507511 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:43.507529 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:43.507625 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:43.507891 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:43.507894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:43.508088 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:43.508195 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:43.508201 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.508982 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.508985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:43.509544 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:43.509553 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.510024 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.510027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:43.510123 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:43.510718 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:43.510730 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:43.513030 (+ 2300us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 07:25:43.513309 27670 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:43.507465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:34409 (local address 127.12.157.254:37163)
0810 07:25:43.507592 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:43.507596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:43.507610 (+ 14us) server_negotiation.cc:407] Connection header received
0810 07:25:43.507738 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:43.507741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:43.507783 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:43.507871 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:43.508322 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.508837 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.509674 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.509903 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.510018 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:43.513176 (+ 3158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.194:34409: BlockingRecv error: recv got EOF from 127.12.157.194:34409 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 07:25:43 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:43 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:43.519731 27884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:43.513819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.194:52839)
0810 07:25:43.513960 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:43.513976 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:43.514063 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:43.514357 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:43.514360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:43.514547 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:43.514652 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:43.514658 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.515379 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.515385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:43.516272 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:43.516283 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.516772 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.516776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:43.516859 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:43.517362 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:43.517377 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:43.519599 (+ 2222us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 07:25:43.519899 27670 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:43.513875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:52839 (local address 127.12.157.254:37163)
0810 07:25:43.513993 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:43.513996 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:43.514015 (+ 19us) server_negotiation.cc:407] Connection header received
0810 07:25:43.514196 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:43.514199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:43.514243 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:43.514317 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:43.514766 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.515268 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.516419 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.516645 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.516735 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:43.519735 (+ 3000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.194:52839: BlockingRecv error: recv got EOF from 127.12.157.194:52839 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 07:25:43.520154 27881 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 07:25:43.614811 27883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:43.615113 27883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:43.615190 27883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:43.619164 27883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:43.619263 27883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:43.619354 27883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:25:43.624596 27883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:42423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/krb5kdc/oryx_127.12.157.195.keytab
--principal=oryx/127.12.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:37699
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=42899
--tserver_master_addrs=127.12.157.254:37163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:43.625726 27883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:43.626767 27883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:43.637140 27896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:43.637293 27883 server_base.cc:1034] running on GCE node
W20260810 07:25:43.637149 27890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:43.637204 27889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:43.638036 27883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:43.638823 27883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:43.640044 27883 hybrid_clock.cc:648] HybridClock initialized: now 1786346743640036 us; error 41 us; skew 500 ppm
Aug 10 07:25:43 dist-test-slave-ffrd krb5kdc[26535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346743, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:25:43.643116 27883 init.cc:377] Logged in from keytab as oryx/127.12.157.195@KRBTEST.COM (short username oryx)
I20260810 07:25:43.644064 27883 webserver.cc:460] Webserver started at http://127.12.157.195:42899/ using document root <none> and password file <none>
I20260810 07:25:43.644663 27883 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:43.644778 27883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:43.648794 27883 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260810 07:25:43.651023 27907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:43.652284 27883 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 07:25:43.652441 27883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "e9f7a12ec4164a339a72ac228cfb4a32"
format_stamp: "Formatted at 2026-08-10 07:25:42 on dist-test-slave-ffrd"
server_key: "9be803fb953a2e3453f23ad3b4c126fc"
server_key_iv: "35e91c09b5e2435d8db29a248ffc3bc7"
server_key_version: "encryptionkey@0"
I20260810 07:25:43.653070 27883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:43.668074 27883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:43.671103 27883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:43.671317 27883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:43.671939 27883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:43.672802 27883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:43.672878 27883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:43.673002 27883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:43.673048 27883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:43.682968 27883 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:37699
I20260810 07:25:43.682992 28082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:37699 every 8 connection(s)
I20260810 07:25:43.684005 27883 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:25:43.691425 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 27883
Aug 10 07:25:43 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:43 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 07:25:43.695518 27670 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:43.685901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:47631 (local address 127.12.157.254:37163)
0810 07:25:43.686045 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:43.686049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:43.686908 (+ 859us) server_negotiation.cc:407] Connection header received
0810 07:25:43.688191 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:43.688194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:43.688255 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:43.688366 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:43.689606 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.690136 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.691303 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.691527 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.691648 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:43.695415 (+ 3767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.195:47631: BlockingRecv error: recv got EOF from 127.12.157.195:47631 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 07:25:43.696272 28091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:43.686274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.195:47631)
0810 07:25:43.686740 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:43.686783 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:43.687975 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:43.688559 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:43.688566 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:43.688961 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:43.689445 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:43.689459 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.690269 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.690273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:43.691166 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:43.691177 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.691725 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.691730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:43.691944 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:43.692552 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:43.692569 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:43.695314 (+ 2745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":128,"threads_started":1}
W20260810 07:25:43.696755 28084 heartbeater.cc:644] Failed to heartbeat to 127.12.157.254:37163 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.157.254:37163: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Aug 10 07:25:43 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:43 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:43.702459 28091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:43.697245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.195:53633)
0810 07:25:43.697380 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:43.697396 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:43.697472 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:43.697687 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:43.697694 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:43.697872 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:43.698007 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:43.698015 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.698695 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.698702 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:43.699418 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:43.699428 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.699970 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.699972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:43.700038 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:43.700463 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:43.700475 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:43.702318 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 07:25:43.702556 27670 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:43.697377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:53633 (local address 127.12.157.254:37163)
0810 07:25:43.697500 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:43.697503 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:43.697513 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:25:43.697542 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:43.697544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:43.697575 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:43.697638 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:43.698122 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.698588 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.699538 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.699849 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.699977 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:43.702449 (+ 2472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.195:53633: BlockingRecv error: recv got EOF from 127.12.157.195:53633 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 07:25:43.702713 28094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:43.692531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52900 (local address 127.12.157.254:37163)
0810 07:25:43.692762 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:43.692765 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:43.693287 (+ 522us) server_negotiation.cc:407] Connection header received
0810 07:25:43.693419 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:43.693423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:43.693485 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:43.693578 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:43.694518 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.695015 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.696142 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.696359 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.696482 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:43.698247 (+ 1765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:43.698253 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:43.698267 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:25:43.698293 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:43.700211 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:43.700667 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:43.700673 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:43.700679 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:43.700804 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:25:43.701157 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:25:43.701161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:25:43.701163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:25:43.701705 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:43.701930 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:43.702222 (+ 292us) server_negotiation.cc:299] Negotiation successful
0810 07:25:43.702480 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":93,"threads_started":1}
Aug 10 07:25:43 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:43 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:43.708738 28091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:43.703164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.195:43017)
0810 07:25:43.703290 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:43.703304 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:43.703376 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:43.703583 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:43.703586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:43.703735 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:43.703874 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:43.703882 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.704860 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.704863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:43.705643 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:43.705654 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.706209 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.706212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:43.706279 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:43.706821 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:43.706834 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:43.708641 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260810 07:25:43.709069 28094 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:43.703273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:43017 (local address 127.12.157.254:37163)
0810 07:25:43.703397 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:43.703401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:43.703420 (+ 19us) server_negotiation.cc:407] Connection header received
0810 07:25:43.703448 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:43.703450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:43.703481 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:43.703544 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:43.704011 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.704706 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.705757 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:43.706105 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:43.706188 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:43.708773 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.195:43017: BlockingRecv error: recv got EOF from 127.12.157.195:43017 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260810 07:25:43.709061 28084 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 07:25:44 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:44 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:44.321400 28095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:44.315466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.193:41685)
0810 07:25:44.315812 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:44.315827 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:44.315967 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:44.316289 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:44.316292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:44.316550 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:44.316686 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:44.316692 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:44.317533 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:44.317536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:44.318122 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:44.318130 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:44.318552 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:44.318555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:44.318619 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:44.319231 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:44.319251 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:44.321276 (+ 2025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":96,"threads_started":1}
W20260810 07:25:44.321605 28098 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:44.315561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:41685 (local address 127.12.157.254:37163)
0810 07:25:44.315885 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:44.315890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:44.315910 (+ 20us) server_negotiation.cc:407] Connection header received
0810 07:25:44.316085 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:44.316088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:44.316146 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:44.316259 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:44.316833 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:44.317431 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:44.318227 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:44.318453 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:44.318529 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:44.321431 (+ 2902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.193:41685: BlockingRecv error: recv got EOF from 127.12.157.193:41685 (error 108)
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":96,"threads_started":1}
Aug 10 07:25:44 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:44 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:44.526387 28099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:44.520800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.194:33129)
0810 07:25:44.521105 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:44.521118 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:44.521225 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:44.521487 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:44.521489 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:44.521680 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:44.521809 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:44.521814 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:44.522533 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:44.522535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:44.523120 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:44.523128 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:44.523585 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:44.523588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:44.523670 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:44.524291 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:44.524304 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:44.526253 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":81,"threads_started":1}
W20260810 07:25:44.526559 28098 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:44.520901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:33129 (local address 127.12.157.254:37163)
0810 07:25:44.521110 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:44.521114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:44.521173 (+ 59us) server_negotiation.cc:407] Connection header received
0810 07:25:44.521310 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:44.521313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:44.521362 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:44.521440 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:44.521912 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:44.522434 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:44.523230 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:44.523453 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:44.523534 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:44.526402 (+ 2868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.194:33129: BlockingRecv error: recv got EOF from 127.12.157.194:33129 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 10 07:25:44 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:44 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:44.715564 28101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:44.709794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.195:37079)
0810 07:25:44.710103 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:44.710117 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:44.710221 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:44.710519 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:44.710522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:44.710710 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:44.710843 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:44.710849 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:44.711633 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:44.711635 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:44.712217 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:44.712226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:44.712678 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:44.712681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:44.712766 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:44.713368 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:44.713381 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:44.715372 (+ 1991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":120,"threads_started":1}
W20260810 07:25:44.715621 28098 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:44.709851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:37079 (local address 127.12.157.254:37163)
0810 07:25:44.710007 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:44.710012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:44.710199 (+ 187us) server_negotiation.cc:407] Connection header received
0810 07:25:44.710306 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:44.710309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:44.710359 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:44.710435 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:44.710948 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:44.711511 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:44.712333 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:44.712573 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:44.712659 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:44.715503 (+ 2844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.195:37079: BlockingRecv error: recv got EOF from 127.12.157.195:37079 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 07:25:45 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:45 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:45.327670 28102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:45.322205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.193:43363)
0810 07:25:45.322538 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:45.322552 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:45.322644 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:45.322895 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:45.322897 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:45.323090 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:45.323192 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:45.323197 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:45.324022 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:45.324025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:45.324581 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:45.324589 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:45.325126 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:45.325128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:45.325192 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:45.325676 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:45.325687 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:45.327527 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":106,"threads_started":1}
W20260810 07:25:45.327883 28103 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:45.322342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:43363 (local address 127.12.157.254:37163)
0810 07:25:45.322623 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:45.322627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:45.322641 (+ 14us) server_negotiation.cc:407] Connection header received
0810 07:25:45.322734 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:45.322737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:45.322781 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:45.322859 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:45.323335 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:45.323906 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:45.324699 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:45.325007 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:45.325111 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:45.327673 (+ 2562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.193:43363: BlockingRecv error: recv got EOF from 127.12.157.193:43363 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":110,"threads_started":1}
Aug 10 07:25:45 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:45 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:45.534144 28113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:45.527381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.194:46571)
0810 07:25:45.527749 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:45.527768 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:45.527928 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:45.528619 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:45.528622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:45.529027 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:45.529151 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:45.529158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:45.529962 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:45.529965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:45.530567 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:45.530575 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:45.531062 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:45.531064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:45.531131 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:45.531709 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:45.531722 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:45.534005 (+ 2283us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":155,"threads_started":1}
W20260810 07:25:45.534336 28103 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:45.527627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:46571 (local address 127.12.157.254:37163)
0810 07:25:45.527786 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:45.527791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:45.527864 (+ 73us) server_negotiation.cc:407] Connection header received
0810 07:25:45.528047 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:45.528049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:45.528348 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:45.528447 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:45.529300 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:45.529841 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:45.530700 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:45.530947 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:45.531037 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:45.534162 (+ 3125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.194:46571: BlockingRecv error: recv got EOF from 127.12.157.194:46571 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 10 07:25:45 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:45 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:45.722950 28115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:45.716690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.195:33573)
0810 07:25:45.717045 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:45.717059 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:45.717166 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:45.717403 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:45.717405 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:45.717597 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:45.717698 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:45.717703 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:45.718522 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:45.718526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:45.719102 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:45.719111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:45.719641 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:45.719644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:45.719711 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:45.720291 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:45.720304 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:45.722764 (+ 2460us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":120,"threads_started":1}
W20260810 07:25:45.723312 28103 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:45.716880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:33573 (local address 127.12.157.254:37163)
0810 07:25:45.717070 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:45.717074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:45.717115 (+ 41us) server_negotiation.cc:407] Connection header received
0810 07:25:45.717236 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:45.717239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:45.717284 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:45.717380 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:45.717812 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:45.718344 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:45.719262 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:45.719516 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:45.719608 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:45.722984 (+ 3376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.195:33573: BlockingRecv error: recv got EOF from 127.12.157.195:33573 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Aug 10 07:25:46 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:46 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:46.334853 28118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:46.328645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.193:55451)
0810 07:25:46.328988 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:46.329009 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:46.329103 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:46.329438 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:46.329441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:46.329634 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:46.329741 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:46.329747 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:46.330725 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:46.330730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:46.331569 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:46.331581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:46.332018 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:46.332021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:46.332081 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:46.332615 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:46.332627 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:46.334731 (+ 2104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":82,"threads_started":1}
W20260810 07:25:46.335083 28123 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:46.328746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:55451 (local address 127.12.157.254:37163)
0810 07:25:46.329061 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:46.329065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:46.329081 (+ 16us) server_negotiation.cc:407] Connection header received
0810 07:25:46.329203 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:46.329205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:46.329251 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:46.329348 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:46.329909 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:46.330575 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:46.331679 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:46.331908 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:46.331989 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:46.334922 (+ 2933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.193:55451: BlockingRecv error: recv got EOF from 127.12.157.193:55451 (error 108)
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":130,"threads_started":1}
Aug 10 07:25:46 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:46 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:46.544643 28124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:46.534918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.194:54025)
0810 07:25:46.535256 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:46.535271 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:46.535376 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:46.535667 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:46.535670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:46.535849 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:46.535965 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:46.535970 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:46.536725 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:46.536727 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:46.537451 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:46.537465 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:46.537927 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:46.537930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:46.537990 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:46.542482 (+ 4492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:46.542495 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:46.544474 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":115,"threads_started":1}
W20260810 07:25:46.544839 28123 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:46.535079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:54025 (local address 127.12.157.254:37163)
0810 07:25:46.535230 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:46.535234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:46.535354 (+ 120us) server_negotiation.cc:407] Connection header received
0810 07:25:46.535497 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:46.535500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:46.535547 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:46.535620 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:46.536085 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:46.536627 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:46.537583 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:46.537800 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:46.537889 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:46.544667 (+ 6778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.194:54025: BlockingRecv error: recv got EOF from 127.12.157.194:54025 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 07:25:46 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:46 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:46.730681 28125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:46.723873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.195:35887)
0810 07:25:46.724170 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:46.724187 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:46.724312 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:46.724655 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:46.724660 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:46.724926 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:46.725122 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:46.725134 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:46.726229 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:46.726238 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:46.726821 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:46.726829 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:46.727383 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:46.727386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:46.727507 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:46.728114 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:46.728136 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:46.730500 (+ 2364us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":116,"threads_started":1}
W20260810 07:25:46.730830 28123 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:46.724285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:35887 (local address 127.12.157.254:37163)
0810 07:25:46.724412 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:46.724416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:46.724438 (+ 22us) server_negotiation.cc:407] Connection header received
0810 07:25:46.724473 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:46.724475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:46.724520 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:46.724587 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:46.725299 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:46.726064 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:46.726930 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:46.727243 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:46.727355 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:46.730680 (+ 3325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.195:35887: BlockingRecv error: recv got EOF from 127.12.157.195:35887 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 10 07:25:47 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:47 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:47.346951 28127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:47.335761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.193:42871)
0810 07:25:47.336636 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:47.336657 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:47.336782 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:47.338122 (+ 1340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:47.338130 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:47.338447 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:47.338595 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:47.338605 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:47.339731 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:47.339734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:47.340296 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:47.340305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:47.340848 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:47.340852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:47.340927 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:47.344617 (+ 3690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:47.344631 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:47.346819 (+ 2188us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":89,"threads_started":1}
W20260810 07:25:47.347167 28128 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:47.335870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:42871 (local address 127.12.157.254:37163)
0810 07:25:47.337808 (+ 1938us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:47.337813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:47.337825 (+ 12us) server_negotiation.cc:407] Connection header received
0810 07:25:47.337869 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:47.337874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:47.337929 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:47.338024 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:47.338737 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:47.339613 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:47.340459 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:47.340691 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:47.340784 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:47.346996 (+ 6212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.193:42871: BlockingRecv error: recv got EOF from 127.12.157.193:42871 (error 108)
Metrics: {"server-negotiator.queue_time_us":1856,"thread_start_us":84,"threads_started":1}
Aug 10 07:25:47 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:47 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:47.551538 28129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:47.545661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.194:34527)
0810 07:25:47.545966 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:47.545979 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:47.546081 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:47.546314 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:47.546316 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:47.546510 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:47.546616 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:47.546622 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:47.547437 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:47.547440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:47.548010 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:47.548019 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:47.548520 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:47.548523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:47.548605 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:47.549209 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:47.549221 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:47.551354 (+ 2133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":123,"threads_started":1}
W20260810 07:25:47.551608 28128 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:47.545731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:34527 (local address 127.12.157.254:37163)
0810 07:25:47.545867 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:47.545872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:47.546049 (+ 177us) server_negotiation.cc:407] Connection header received
0810 07:25:47.546148 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:47.546151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:47.546195 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:47.546264 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:47.546750 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:47.547290 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:47.548143 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:47.548374 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:47.548483 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:47.551496 (+ 3013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.194:34527: BlockingRecv error: recv got EOF from 127.12.157.194:34527 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 07:25:47 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:47 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:47.738098 28131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:47.731916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.195:48553)
0810 07:25:47.732240 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:47.732257 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:47.732393 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:47.732694 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:47.732697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:47.732976 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:47.733079 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:47.733086 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:47.733876 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:47.733882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:47.734463 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:47.734472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:47.734962 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:47.734965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:47.735062 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:47.735613 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:47.735625 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:47.737916 (+ 2291us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":96,"threads_started":1}
W20260810 07:25:47.738171 28128 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:47.732054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:48553 (local address 127.12.157.254:37163)
0810 07:25:47.732240 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:47.732247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:47.732323 (+ 76us) server_negotiation.cc:407] Connection header received
0810 07:25:47.732514 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:47.732517 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:47.732574 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:47.732664 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:47.733209 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:47.733754 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:47.734606 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:47.734832 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:47.734946 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:47.738058 (+ 3112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.195:48553: BlockingRecv error: recv got EOF from 127.12.157.195:48553 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 10 07:25:48 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:48 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:48.353703 28132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:48.347875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.193:58567)
0810 07:25:48.348164 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:48.348178 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:48.348266 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:48.348667 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:48.348670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:48.348870 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:48.349040 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:48.349047 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:48.349864 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:48.349867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:48.350472 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:48.350481 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:48.350940 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:48.350943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:48.351008 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:48.351542 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:48.351555 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:48.353577 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":102,"threads_started":1}
W20260810 07:25:48.353916 28137 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:48.348120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:58567 (local address 127.12.157.254:37163)
0810 07:25:48.348405 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:48.348409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:48.348422 (+ 13us) server_negotiation.cc:407] Connection header received
0810 07:25:48.348458 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:48.348460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:48.348502 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:48.348586 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:48.349169 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:48.349743 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:48.350591 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:48.350822 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:48.350911 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:48.353730 (+ 2819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.193:58567: BlockingRecv error: recv got EOF from 127.12.157.193:58567 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":123,"threads_started":1}
Aug 10 07:25:48 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:48 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:48.558627 28138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:48.552542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.194:60275)
0810 07:25:48.552855 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:48.552871 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:48.552999 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:48.553242 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:48.553244 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:48.553444 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:48.553544 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:48.553549 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:48.554331 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:48.554351 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:48.554947 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:48.554956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:48.555387 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:48.555389 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:48.555451 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:48.556084 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:48.556097 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:48.558499 (+ 2402us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":118,"threads_started":1}
W20260810 07:25:48.558792 28137 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:48.552659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:60275 (local address 127.12.157.254:37163)
0810 07:25:48.552850 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:48.552853 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:48.552960 (+ 107us) server_negotiation.cc:407] Connection header received
0810 07:25:48.553081 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:48.553083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:48.553129 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:48.553203 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:48.553667 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:48.554216 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:48.555060 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:48.555279 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:48.555372 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:48.558659 (+ 3287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.194:60275: BlockingRecv error: recv got EOF from 127.12.157.194:60275 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 10 07:25:48 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:48 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:48.745155 28140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:48.738998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.195:50655)
0810 07:25:48.739360 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:48.739380 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:48.739525 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:48.739825 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:48.739828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:48.740058 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:48.740154 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:48.740160 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:48.740962 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:48.740967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:48.741586 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:48.741595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:48.742092 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:48.742095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:48.742188 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:48.742759 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:48.742772 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:48.744951 (+ 2179us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":111,"threads_started":1}
W20260810 07:25:48.745216 28137 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:48.739127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:50655 (local address 127.12.157.254:37163)
0810 07:25:48.739301 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:48.739307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:48.739469 (+ 162us) server_negotiation.cc:407] Connection header received
0810 07:25:48.739642 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:48.739646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:48.739706 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:48.739802 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:48.740288 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:48.740838 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:48.741723 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:48.741957 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:48.742084 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:48.745104 (+ 3020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.195:50655: BlockingRecv error: recv got EOF from 127.12.157.195:50655 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 10 07:25:49 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:49 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:49.360916 28142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:49.354602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.193:52461)
0810 07:25:49.354922 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:49.354943 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:49.355050 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:49.355416 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:49.355419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:49.355648 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:49.355787 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:49.355795 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:49.356865 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:49.356868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:49.357658 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:49.357669 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:49.358140 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:49.358143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:49.358212 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:49.358769 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:49.358796 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:49.360793 (+ 1997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":78,"threads_started":1}
W20260810 07:25:49.361158 28143 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:49.354759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:52461 (local address 127.12.157.254:37163)
0810 07:25:49.355132 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:49.355137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:49.355154 (+ 17us) server_negotiation.cc:407] Connection header received
0810 07:25:49.355203 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:49.355208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:49.355265 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:49.355393 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:49.355915 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:49.356747 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:49.357779 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:49.358005 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:49.358089 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:49.360969 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.193:52461: BlockingRecv error: recv got EOF from 127.12.157.193:52461 (error 108)
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":104,"threads_started":1}
Aug 10 07:25:49 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:49 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:49.565850 28145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:49.559482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.194:53655)
0810 07:25:49.560021 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:49.560037 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:49.560166 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:49.560537 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:49.560540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:49.560741 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:49.560845 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:49.560851 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:49.561791 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:49.561796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:49.562437 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:49.562483 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:49.562969 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:49.562973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:49.563059 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:49.563653 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:49.563666 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:49.565729 (+ 2063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":329,"threads_started":1}
W20260810 07:25:49.566138 28143 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:49.559882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:53655 (local address 127.12.157.254:37163)
0810 07:25:49.560125 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:49.560132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:49.560151 (+ 19us) server_negotiation.cc:407] Connection header received
0810 07:25:49.560314 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:49.560317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:49.560369 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:49.560459 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:49.561037 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:49.561671 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:49.562605 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:49.562843 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:49.562959 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:49.565905 (+ 2946us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.194:53655: BlockingRecv error: recv got EOF from 127.12.157.194:53655 (error 108)
Metrics: {"server-negotiator.queue_time_us":139}
Aug 10 07:25:49 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:49 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:49.752225 28148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:49.746236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.195:48295)
0810 07:25:49.746709 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:49.746724 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:49.746841 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:49.747110 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:49.747113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:49.747350 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:49.747469 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:49.747479 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:49.748310 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:49.748313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:49.748933 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:49.748972 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:49.749503 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:49.749507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:49.749581 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:49.750152 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:49.750166 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:49.752091 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":157,"threads_started":1}
W20260810 07:25:49.752439 28143 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:49.746331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:48295 (local address 127.12.157.254:37163)
0810 07:25:49.746493 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:49.746498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:49.746808 (+ 310us) server_negotiation.cc:407] Connection header received
0810 07:25:49.746929 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:49.746931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:49.746981 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:49.747080 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:49.747627 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:49.748173 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:49.749095 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:49.749330 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:49.749455 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:49.752257 (+ 2802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.195:48295: BlockingRecv error: recv got EOF from 127.12.157.195:48295 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 10 07:25:50 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:50 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:50.367295 28149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:50.361834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.193:35477)
0810 07:25:50.362204 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:50.362218 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:50.362314 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:50.362537 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:50.362538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:50.362727 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:50.362821 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:50.362826 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:50.363591 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:50.363593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:50.364154 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:50.364162 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:50.364586 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:50.364588 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:50.364658 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:50.365223 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:50.365235 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:50.367178 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":88,"threads_started":1}
W20260810 07:25:50.367543 28151 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:50.361910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:35477 (local address 127.12.157.254:37163)
0810 07:25:50.362213 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:50.362217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:50.362264 (+ 47us) server_negotiation.cc:407] Connection header received
0810 07:25:50.362401 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:50.362404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:50.362449 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:50.362526 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:50.362927 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:50.363488 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:50.364259 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:50.364477 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:50.364553 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:50.367373 (+ 2820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.193:35477: BlockingRecv error: recv got EOF from 127.12.157.193:35477 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":106,"threads_started":1}
Aug 10 07:25:50 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:50 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:50.572506 28153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:50.566713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.194:48023)
0810 07:25:50.567037 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:50.567051 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:50.567157 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:50.567432 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:50.567434 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:50.567646 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:50.567747 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:50.567752 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:50.568528 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:50.568530 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:50.569231 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:50.569241 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:50.569749 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:50.569753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:50.569826 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:50.570373 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:50.570387 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:50.572383 (+ 1996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":95,"threads_started":1}
W20260810 07:25:50.572711 28151 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:50.566830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:48023 (local address 127.12.157.254:37163)
0810 07:25:50.566985 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:50.566989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:50.567126 (+ 137us) server_negotiation.cc:407] Connection header received
0810 07:25:50.567224 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:50.567227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:50.567272 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:50.567344 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:50.567862 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:50.568416 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:50.569371 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:50.569613 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:50.569705 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:50.572566 (+ 2861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.194:48023: BlockingRecv error: recv got EOF from 127.12.157.194:48023 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 07:25:50 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:50 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:50.759918 28155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:50.753227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.195:46237)
0810 07:25:50.753557 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:50.753571 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:50.753674 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:50.754025 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:50.754029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:50.754257 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:50.754510 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:50.754529 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:50.755666 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:50.755669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:50.756357 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:50.756367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:50.756866 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:50.756869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:50.757016 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:50.757761 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:50.757776 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:50.759774 (+ 1998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":104,"threads_started":1}
W20260810 07:25:50.760159 28151 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:50.753356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:46237 (local address 127.12.157.254:37163)
0810 07:25:50.753509 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:50.753513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:50.753691 (+ 178us) server_negotiation.cc:407] Connection header received
0810 07:25:50.753751 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:50.753755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:50.753838 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:50.753954 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:50.754686 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:50.755534 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:50.756505 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:50.756725 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:50.756816 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:50.759930 (+ 3114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.195:46237: BlockingRecv error: recv got EOF from 127.12.157.195:46237 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 07:25:51 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:51 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:51.373742 28156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:51.368051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.193:36085)
0810 07:25:51.368360 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:51.368377 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:51.368499 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:51.368865 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:51.368870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:51.369139 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:51.369295 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:51.369305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:51.370189 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:51.370192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:51.370747 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:51.370755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:51.371159 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:51.371162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:51.371222 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:51.371727 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:51.371739 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:51.373626 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":130,"threads_started":1}
W20260810 07:25:51.374023 28160 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:51.368208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:36085 (local address 127.12.157.254:37163)
0810 07:25:51.368438 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:51.368441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:51.368454 (+ 13us) server_negotiation.cc:407] Connection header received
0810 07:25:51.368629 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:51.368633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:51.368694 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:51.368803 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:51.369454 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:51.370090 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:51.370847 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:51.371065 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:51.371139 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:51.373762 (+ 2623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.193:36085: BlockingRecv error: recv got EOF from 127.12.157.193:36085 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":85,"threads_started":1}
Aug 10 07:25:51 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:51 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:51.579560 28161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:51.573506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.194:37873)
0810 07:25:51.573801 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:51.573815 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:51.573952 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:51.574245 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:51.574247 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:51.574448 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:51.574558 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:51.574564 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:51.575518 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:51.575521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:51.576226 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:51.576237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:51.576706 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:51.576709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:51.576782 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:51.577394 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:51.577406 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:51.579434 (+ 2028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":112,"threads_started":1}
W20260810 07:25:51.579753 28160 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:51.573572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:37873 (local address 127.12.157.254:37163)
0810 07:25:51.573748 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:51.573754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:51.573889 (+ 135us) server_negotiation.cc:407] Connection header received
0810 07:25:51.574073 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:51.574076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:51.574129 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:51.574212 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:51.574680 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:51.575360 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:51.576344 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:51.576581 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:51.576658 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:51.579615 (+ 2957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.194:37873: BlockingRecv error: recv got EOF from 127.12.157.194:37873 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 07:25:51 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:51 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:51.766474 28162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:51.760737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.195:45109)
0810 07:25:51.761147 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:51.761163 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:51.761285 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:51.761596 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:51.761599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:51.761809 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:51.761912 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:51.761918 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:51.762724 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:51.762729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:51.763289 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:51.763298 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:51.763785 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:51.763789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:51.763875 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:51.764366 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:51.764378 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:51.766285 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":126,"threads_started":1}
W20260810 07:25:51.766510 28160 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:51.760798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:45109 (local address 127.12.157.254:37163)
0810 07:25:51.760971 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:51.760978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:51.761257 (+ 279us) server_negotiation.cc:407] Connection header received
0810 07:25:51.761412 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:51.761415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:51.761475 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:51.761563 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:51.762074 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:51.762606 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:51.763424 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:51.763665 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:51.763776 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:51.766412 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.195:45109: BlockingRecv error: recv got EOF from 127.12.157.195:45109 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 10 07:25:52 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:52 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:52.384492 28163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:52.374575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.193:44863)
0810 07:25:52.374869 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:52.374883 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:52.375015 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:52.375346 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:52.375350 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:52.375600 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:52.375704 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:52.375711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:52.376486 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:52.376489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:52.377143 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:52.377153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:52.377642 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:52.377644 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:52.377708 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:52.382132 (+ 4424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:52.382145 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:52.384330 (+ 2185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":110,"threads_started":1}
W20260810 07:25:52.384635 28164 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:52.374673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:44863 (local address 127.12.157.254:37163)
0810 07:25:52.374941 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:52.374945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:52.374958 (+ 13us) server_negotiation.cc:407] Connection header received
0810 07:25:52.375129 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:52.375132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:52.375192 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:52.375295 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:52.375847 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:52.376386 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:52.377282 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:52.377530 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:52.377622 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:52.384515 (+ 6893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.193:44863: BlockingRecv error: recv got EOF from 127.12.157.193:44863 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":75,"threads_started":1}
Aug 10 07:25:52 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:52 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:52.586722 28166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:52.580532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.194:40901)
0810 07:25:52.580933 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:52.580975 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:52.581121 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:52.581411 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:52.581414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:52.581638 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:52.581736 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:52.581741 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:52.582530 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:52.582535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:52.583103 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:52.583111 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:52.583588 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:52.583592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:52.583681 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:52.584255 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:52.584268 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:52.586582 (+ 2314us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":96,"threads_started":1}
W20260810 07:25:52.586942 28164 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:52.580683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:40901 (local address 127.12.157.254:37163)
0810 07:25:52.580874 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:52.580879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:52.581058 (+ 179us) server_negotiation.cc:407] Connection header received
0810 07:25:52.581227 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:52.581231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:52.581285 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:52.581383 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:52.581864 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:52.582414 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:52.583228 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:52.583466 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:52.583581 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:52.586733 (+ 3152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.194:40901: BlockingRecv error: recv got EOF from 127.12.157.194:40901 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 10 07:25:52 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:52 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:52.772991 28169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:52.767320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.195:55179)
0810 07:25:52.767605 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:52.767618 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:52.767723 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:52.767975 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:52.767977 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:52.768174 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:52.768281 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:52.768286 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:52.769069 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:52.769076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:52.769733 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:52.769758 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:52.770229 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:52.770232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:52.770300 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:52.770829 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:52.770842 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:52.772804 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":107,"threads_started":1}
W20260810 07:25:52.773061 28164 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:52.767456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:55179 (local address 127.12.157.254:37163)
0810 07:25:52.767597 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:52.767601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:52.767675 (+ 74us) server_negotiation.cc:407] Connection header received
0810 07:25:52.767813 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:52.767816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:52.767863 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:52.767926 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:52.768385 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:52.768914 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:52.769868 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:52.770092 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:52.770175 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:52.772965 (+ 2790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.195:55179: BlockingRecv error: recv got EOF from 127.12.157.195:55179 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 07:25:53 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:53 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:53.392853 28179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:53.385952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.193:47045)
0810 07:25:53.386419 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:53.386437 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:53.386568 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:53.386838 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:53.386841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:53.387055 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:53.387162 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:53.387168 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:53.388371 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:53.388375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:53.389114 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:53.389131 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:53.389658 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:53.389661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:53.389736 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:53.390362 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:53.390375 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:53.392652 (+ 2277us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":209,"threads_started":1}
W20260810 07:25:53.392978 28175 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:53.386022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:47045 (local address 127.12.157.254:37163)
0810 07:25:53.386331 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:53.386335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:53.386563 (+ 228us) server_negotiation.cc:407] Connection header received
0810 07:25:53.386642 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:53.386644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:53.386692 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:53.386781 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:53.387361 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:53.388212 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:53.389251 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:53.389521 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:53.389608 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:53.392810 (+ 3202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.193:47045: BlockingRecv error: recv got EOF from 127.12.157.193:47045 (error 108)
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":120,"threads_started":1}
Aug 10 07:25:53 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:53 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:53.593961 28181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:53.587565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.194:49941)
0810 07:25:53.587952 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:53.587965 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:53.588070 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:53.588304 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:53.588307 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:53.588508 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:53.588614 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:53.588619 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:53.589651 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:53.589654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:53.590233 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:53.590242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:53.590708 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:53.590711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:53.590812 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:53.591433 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:53.591445 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:53.593762 (+ 2317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":120,"threads_started":1}
W20260810 07:25:53.594027 28175 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:53.587805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:49941 (local address 127.12.157.254:37163)
0810 07:25:53.587958 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:53.587962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:53.588021 (+ 59us) server_negotiation.cc:407] Connection header received
0810 07:25:53.588138 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:53.588140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:53.588181 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:53.588247 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:53.588736 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:53.589506 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:53.590344 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:53.590575 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:53.590658 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:53.593898 (+ 3240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.194:49941: BlockingRecv error: recv got EOF from 127.12.157.194:49941 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 10 07:25:53 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:53 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:53.779883 28182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:53.773931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.195:42071)
0810 07:25:53.774298 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:53.774315 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:53.774457 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:53.774755 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:53.774758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:53.774993 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:53.775090 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:53.775120 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:53.775905 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:53.775910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:53.776582 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:53.776593 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:53.777143 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:53.777147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:53.777242 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:53.777791 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:53.777804 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:53.779766 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":122,"threads_started":1}
W20260810 07:25:53.780041 28175 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:53.774091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:42071 (local address 127.12.157.254:37163)
0810 07:25:53.774282 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:53.774287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:53.774406 (+ 119us) server_negotiation.cc:407] Connection header received
0810 07:25:53.774577 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:53.774580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:53.774641 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:53.774734 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:53.775221 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:53.775787 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:53.776711 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:53.777016 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:53.777134 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:53.779902 (+ 2768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.195:42071: BlockingRecv error: recv got EOF from 127.12.157.195:42071 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 10 07:25:54 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:54 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:54.400046 28183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:54.393903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.193:59987)
0810 07:25:54.394248 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:54.394263 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:54.394381 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:54.394745 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:54.394748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:54.394949 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:54.395053 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:54.395059 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:54.395891 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:54.395898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:54.396563 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:54.396578 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:54.397211 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:54.397215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:54.397314 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:54.397963 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:54.397975 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:54.399855 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":94,"threads_started":1}
W20260810 07:25:54.400106 28184 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:54.394068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:59987 (local address 127.12.157.254:37163)
0810 07:25:54.394313 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:54.394317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:54.394333 (+ 16us) server_negotiation.cc:407] Connection header received
0810 07:25:54.394531 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:54.394534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:54.394586 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:54.394672 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:54.395177 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:54.395755 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:54.396686 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:54.397071 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:54.397193 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:54.399982 (+ 2789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.193:59987: BlockingRecv error: recv got EOF from 127.12.157.193:59987 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":62,"threads_started":1}
Aug 10 07:25:54 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:54 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:54.601402 28185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:54.594892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.194:42813)
0810 07:25:54.595200 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:54.595215 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:54.595370 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:54.595654 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:54.595657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:54.595855 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:54.595963 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:54.595968 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:54.596834 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:54.596837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:54.597502 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:54.597512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:54.598104 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:54.598108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:54.598227 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:54.598820 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:54.598833 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:54.601220 (+ 2387us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":79,"threads_started":1}
W20260810 07:25:54.601468 28184 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:54.595058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:42813 (local address 127.12.157.254:37163)
0810 07:25:54.595242 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:54.595246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:54.595326 (+ 80us) server_negotiation.cc:407] Connection header received
0810 07:25:54.595461 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:54.595464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:54.595510 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:54.595580 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:54.596112 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:54.596715 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:54.597654 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:54.597899 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:54.598051 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:54.601351 (+ 3300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.194:42813: BlockingRecv error: recv got EOF from 127.12.157.194:42813 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 10 07:25:54 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:54 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:54.786891 28186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:54.780963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.195:45831)
0810 07:25:54.781262 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:54.781275 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:54.781384 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:54.781826 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:54.781828 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:54.782027 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:54.782133 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:54.782139 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:54.783010 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:54.783013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:54.783648 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:54.783659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:54.784165 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:54.784168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:54.784239 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:54.784779 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:54.784791 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:54.786770 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":118,"threads_started":1}
W20260810 07:25:54.787078 28184 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:54.780999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:45831 (local address 127.12.157.254:37163)
0810 07:25:54.781151 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:54.781155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:54.781377 (+ 222us) server_negotiation.cc:407] Connection header received
0810 07:25:54.781622 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:54.781624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:54.781677 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:54.781750 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:54.782251 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:54.782876 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:54.783790 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:54.784029 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:54.784122 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:54.786910 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.195:45831: BlockingRecv error: recv got EOF from 127.12.157.195:45831 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 07:25:55 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:55 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:55.406404 28189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:55.400850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.193:35979)
0810 07:25:55.401206 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:55.401220 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:55.401342 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:55.401636 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:55.401638 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:55.401845 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:55.401950 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:55.401957 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:55.402710 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:55.402713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:55.403287 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:55.403296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:55.403718 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:55.403720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:55.403786 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:55.404308 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:55.404320 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:55.406227 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":133,"threads_started":1}
W20260810 07:25:55.406596 28190 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:55.401031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:35979 (local address 127.12.157.254:37163)
0810 07:25:55.401269 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:55.401273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:55.401286 (+ 13us) server_negotiation.cc:407] Connection header received
0810 07:25:55.401460 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:55.401463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:55.401515 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:55.401594 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:55.402062 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:55.402603 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:55.403395 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:55.403622 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:55.403698 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:55.406414 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.193:35979: BlockingRecv error: recv got EOF from 127.12.157.193:35979 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":61,"threads_started":1}
Aug 10 07:25:55 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:55 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:55.608609 28191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:55.602320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.194:54555)
0810 07:25:55.602708 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:55.602723 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:55.602877 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:55.603180 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:55.603183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:55.603429 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:55.603534 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:55.603540 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:55.604421 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:55.604426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:55.605105 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:55.605115 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:55.605632 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:55.605636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:55.605736 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:55.606291 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:55.606303 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:55.608408 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":121,"threads_started":1}
W20260810 07:25:55.608843 28190 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:55.602454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:54555 (local address 127.12.157.254:37163)
0810 07:25:55.602650 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:55.602655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:55.602810 (+ 155us) server_negotiation.cc:407] Connection header received
0810 07:25:55.602990 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:55.602995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:55.603055 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:55.603153 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:55.603671 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:55.604286 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:55.605243 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:55.605500 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:55.605628 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:55.608555 (+ 2927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.194:54555: BlockingRecv error: recv got EOF from 127.12.157.194:54555 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 10 07:25:55 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:55 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:55.793412 28194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:55.787840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.195:42687)
0810 07:25:55.788159 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:55.788173 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:55.788281 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:55.788565 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:55.788568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:55.788768 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:55.788871 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:55.788876 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:55.789646 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:55.789648 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:55.790214 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:55.790222 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:55.790644 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:55.790647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:55.790713 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:55.791328 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:55.791341 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:55.793251 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":129,"threads_started":1}
W20260810 07:25:55.793605 28190 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:55.787891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:42687 (local address 127.12.157.254:37163)
0810 07:25:55.788051 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:55.788055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:55.788257 (+ 202us) server_negotiation.cc:407] Connection header received
0810 07:25:55.788360 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:55.788363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:55.788421 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:55.788492 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:55.788988 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:55.789541 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:55.790322 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:55.790535 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:55.790612 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:55.793434 (+ 2822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.195:42687: BlockingRecv error: recv got EOF from 127.12.157.195:42687 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 07:25:56 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:56 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:56.413417 28195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:56.407457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.193:58267)
0810 07:25:56.407758 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:56.407771 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:56.407868 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:56.408137 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:56.408139 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:56.408327 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:56.408439 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:56.408445 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:56.409281 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:56.409283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:56.409857 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:56.409866 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:56.410456 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:56.410460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:56.410559 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:56.411268 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:56.411283 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:56.413290 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":120,"threads_started":1}
W20260810 07:25:56.413578 28196 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:56.407591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:58267 (local address 127.12.157.254:37163)
0810 07:25:56.407840 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:56.407844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:56.407859 (+ 15us) server_negotiation.cc:407] Connection header received
0810 07:25:56.407958 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:56.407961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:56.408005 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:56.408098 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:56.408551 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:56.409183 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:56.409995 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:56.410311 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:56.410440 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:56.413430 (+ 2990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.193:58267: BlockingRecv error: recv got EOF from 127.12.157.193:58267 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":91,"threads_started":1}
Aug 10 07:25:56 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:56 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:56.616058 28197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:56.610067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.194:50085)
0810 07:25:56.610417 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:56.610430 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:56.610538 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:56.610919 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:56.610923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:56.611171 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:56.611290 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:56.611298 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:56.612108 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:56.612112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:56.612670 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:56.612678 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:56.613256 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:56.613260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:56.613349 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:56.613933 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:56.613946 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:56.615867 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":150,"threads_started":1}
W20260810 07:25:56.616096 28196 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:56.610238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:50085 (local address 127.12.157.254:37163)
0810 07:25:56.610423 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:56.610427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:56.610507 (+ 80us) server_negotiation.cc:407] Connection header received
0810 07:25:56.610652 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:56.610655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:56.610713 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:56.610817 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:56.611435 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:56.611990 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:56.612812 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:56.613134 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:56.613249 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:56.615996 (+ 2747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.194:50085: BlockingRecv error: recv got EOF from 127.12.157.194:50085 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 10 07:25:56 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:56 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:56.802161 28199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:56.794365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.195:36363)
0810 07:25:56.794659 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:56.794674 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:56.794829 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:56.795261 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:56.795266 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:56.795545 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:56.795689 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:56.795700 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:56.796631 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:56.796637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:56.797784 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:56.797799 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:56.798516 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:56.798522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:56.798645 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:56.799281 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:56.799299 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:56.801942 (+ 2643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":107,"threads_started":1}
W20260810 07:25:56.802192 28196 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:56.794434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:36363 (local address 127.12.157.254:37163)
0810 07:25:56.794591 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:56.794596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:56.794774 (+ 178us) server_negotiation.cc:407] Connection header received
0810 07:25:56.794943 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:56.794946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:56.795012 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:56.795102 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:56.795874 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:56.796528 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:56.797932 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:56.798343 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:56.798470 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:56.802097 (+ 3627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.195:36363: BlockingRecv error: recv got EOF from 127.12.157.195:36363 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 10 07:25:57 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:57 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:57.423751 28200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:57.414350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.193:55455)
0810 07:25:57.414651 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:57.414665 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:57.414761 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:57.415012 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:57.415014 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:57.415208 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:57.415308 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:57.415313 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:57.416065 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:57.416067 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:57.416638 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:57.416645 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:57.417103 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:57.417106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:57.417171 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:57.420526 (+ 3355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:57.420548 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:57.423477 (+ 2929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":104,"threads_started":1}
W20260810 07:25:57.424628 28201 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:57.414425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:55455 (local address 127.12.157.254:37163)
0810 07:25:57.414771 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:57.414775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:57.414810 (+ 35us) server_negotiation.cc:407] Connection header received
0810 07:25:57.414848 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:57.414850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:57.414893 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:57.414963 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:57.415429 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:57.415960 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:57.416741 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:57.416993 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:57.417071 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:57.424180 (+ 7109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.193:55455: BlockingRecv error: recv got EOF from 127.12.157.193:55455 (error 108)
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":69,"threads_started":1}
Aug 10 07:25:57 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:57 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:57.623358 28202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:57.616934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.194:50453)
0810 07:25:57.617250 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:57.617274 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:57.617557 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:57.617888 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:57.617891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:57.618148 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:57.618309 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:57.618318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:57.619309 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:57.619314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:57.619878 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:57.619887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:57.620373 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:57.620377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:57.620466 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:57.621115 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:57.621128 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:57.623159 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":100,"threads_started":1}
W20260810 07:25:57.623402 28201 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:57.617338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:50453 (local address 127.12.157.254:37163)
0810 07:25:57.617480 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:57.617487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:57.617505 (+ 18us) server_negotiation.cc:407] Connection header received
0810 07:25:57.617680 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:57.617684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:57.617737 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:57.617827 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:57.618436 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:57.619196 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:57.620015 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:57.620242 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:57.620356 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:57.623306 (+ 2950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.194:50453: BlockingRecv error: recv got EOF from 127.12.157.194:50453 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 10 07:25:57 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:57 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.195@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:57.809394 28204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:57.803141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.195:50591)
0810 07:25:57.803517 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:57.803538 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:57.803685 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:57.803987 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:57.803990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:57.804218 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:57.804326 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:57.804332 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:57.805279 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:57.805285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:57.805952 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:57.805962 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:57.806621 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:57.806625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:57.806703 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:57.807232 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:57.807244 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:57.809240 (+ 1996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":101,"threads_started":1}
W20260810 07:25:57.809521 28201 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:57.803300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:50591 (local address 127.12.157.254:37163)
0810 07:25:57.803480 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:57.803486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:57.803619 (+ 133us) server_negotiation.cc:407] Connection header received
0810 07:25:57.803803 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:57.803807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:57.803863 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:57.803956 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:57.804528 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:57.805158 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:57.806079 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:57.806392 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:57.806512 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:57.809401 (+ 2889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.195:50591: BlockingRecv error: recv got EOF from 127.12.157.195:50591 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 10 07:25:58 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:58 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.193@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:58.430150 28205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:58.424680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.193:47889)
0810 07:25:58.424994 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:58.425008 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:58.425094 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:58.425368 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:58.425371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:58.425565 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:58.425665 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:58.425671 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:58.426462 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:58.426464 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:58.427020 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:58.427028 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:58.427439 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:58.427442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:58.427504 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:58.428033 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:58.428045 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:58.430032 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":74,"threads_started":1}
W20260810 07:25:58.430336 28206 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:58.424774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:47889 (local address 127.12.157.254:37163)
0810 07:25:58.425109 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:58.425113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:58.425147 (+ 34us) server_negotiation.cc:407] Connection header received
0810 07:25:58.425181 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:58.425184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:58.425226 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:58.425296 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:58.425802 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:58.426351 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:58.427121 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:58.427341 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:58.427419 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:58.430185 (+ 2766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.193:47889: BlockingRecv error: recv got EOF from 127.12.157.193:47889 (error 108)
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":79,"threads_started":1}
Aug 10 07:25:58 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 07:25:58 dist-test-slave-ffrd krb5kdc[26535](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.157.194@KRBTEST.COM for oryx/127.12.157.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 07:25:58.630613 28208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:58.624469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37163 (local address 127.12.157.194:48027)
0810 07:25:58.624791 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:58.624805 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:58.624997 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:58.625287 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:58.625290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:58.625527 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:58.625621 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:58.625626 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:58.626386 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:58.626391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:58.626960 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:58.626968 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:58.627434 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:58.627438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:58.627526 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:58.628139 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:25:58.628151 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:58.630468 (+ 2317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.157.254:37163: Server oryx/127.12.157.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":107,"threads_started":1}
W20260810 07:25:58.630957 28206 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:58.624565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:48027 (local address 127.12.157.254:37163)
0810 07:25:58.624720 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:58.624726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:58.624880 (+ 154us) server_negotiation.cc:407] Connection header received
0810 07:25:58.625122 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:58.625125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:58.625179 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:58.625268 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:58.625737 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:58.626274 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:58.627091 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:58.627319 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:58.627433 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:58.630663 (+ 3230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.157.194:48027: BlockingRecv error: recv got EOF from 127.12.157.194:48027 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 07:25:58.702468 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 27430
I20260810 07:25:58.715507 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 27672
I20260810 07:25:58.725781 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 27883
I20260810 07:25:58.737879 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 27318
2026-08-10T07:25:58Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18389 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T07:25:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:25:58Z Disabled control of system clock
I20260810 07:25:58.775900 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:46283
--webserver_interface=127.12.157.254
--webserver_port=0
--builtin_ntp_servers=127.12.157.212:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:46283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 07:25:58.896257 28213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:58.896545 28213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:58.896581 28213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:58.900696 28213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:25:58.900785 28213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:58.900800 28213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:58.900818 28213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:25:58.900835 28213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:25:58.906041 28213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:46283
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:46283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:58.907294 28213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:58.908771 28213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:58.917609 28229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:58.917613 28225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:58.917627 28226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:58.917677 28213 server_base.cc:1034] running on GCE node
I20260810 07:25:58.918466 28213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:58.919586 28213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:58.920820 28213 hybrid_clock.cc:648] HybridClock initialized: now 1786346758920781 us; error 51 us; skew 500 ppm
I20260810 07:25:58.922770 28213 webserver.cc:460] Webserver started at http://127.12.157.254:32827/ using document root <none> and password file <none>
I20260810 07:25:58.923379 28213 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:58.923478 28213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:58.923717 28213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:58.925843 28213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "dd46c86fb5cc404383cfe9ccc3d1fcca"
format_stamp: "Formatted at 2026-08-10 07:25:58 on dist-test-slave-ffrd"
server_key: "b2744017df37423f82f702f1d82d0e51"
server_key_iv: "643f47748ac5447c4af0401ca383e3c5"
server_key_version: "encryptionkey@0"
I20260810 07:25:58.926415 28213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "dd46c86fb5cc404383cfe9ccc3d1fcca"
format_stamp: "Formatted at 2026-08-10 07:25:58 on dist-test-slave-ffrd"
server_key: "b2744017df37423f82f702f1d82d0e51"
server_key_iv: "643f47748ac5447c4af0401ca383e3c5"
server_key_version: "encryptionkey@0"
I20260810 07:25:58.930039 28213 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260810 07:25:58.932498 28236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:58.933805 28213 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 07:25:58.933957 28213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "dd46c86fb5cc404383cfe9ccc3d1fcca"
format_stamp: "Formatted at 2026-08-10 07:25:58 on dist-test-slave-ffrd"
server_key: "b2744017df37423f82f702f1d82d0e51"
server_key_iv: "643f47748ac5447c4af0401ca383e3c5"
server_key_version: "encryptionkey@0"
I20260810 07:25:58.934074 28213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:58.957050 28213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:58.957814 28213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:58.958025 28213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:58.965487 28213 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:46283
I20260810 07:25:58.965543 28309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:46283 every 8 connection(s)
I20260810 07:25:58.966490 28213 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:25:58.969892 28312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:58.972862 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 28213
I20260810 07:25:58.973013 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:25:58.973295 12919 external_mini_cluster.cc:1428] Setting key 985e6a3df51d6815a8dd28dbf207247b
I20260810 07:25:58.976243 28312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd46c86fb5cc404383cfe9ccc3d1fcca: Bootstrap starting.
I20260810 07:25:58.978519 28312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd46c86fb5cc404383cfe9ccc3d1fcca: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:58.979266 28312 log.cc:824] T 00000000000000000000000000000000 P dd46c86fb5cc404383cfe9ccc3d1fcca: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:58.981320 28312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd46c86fb5cc404383cfe9ccc3d1fcca: No bootstrap required, opened a new log
I20260810 07:25:58.982780 28315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:58.974801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48290 (local address 127.12.157.254:46283)
0810 07:25:58.975471 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:58.975487 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:58.975521 (+ 34us) server_negotiation.cc:407] Connection header received
0810 07:25:58.976181 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:58.976198 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:58.976639 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:58.976997 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:58.977459 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:58.978413 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:58.979351 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:58.979866 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:58.980062 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:58.981031 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:58.981036 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:58.981050 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:25:58.981469 (+ 419us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:58.981543 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:25:58.981565 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:58.981676 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:58.981829 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 07:25:58.982110 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":135,"threads_started":1}
I20260810 07:25:58.985332 28312 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd46c86fb5cc404383cfe9ccc3d1fcca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd46c86fb5cc404383cfe9ccc3d1fcca" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46283 } }
I20260810 07:25:58.985571 28312 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd46c86fb5cc404383cfe9ccc3d1fcca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:58.985658 28312 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd46c86fb5cc404383cfe9ccc3d1fcca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd46c86fb5cc404383cfe9ccc3d1fcca, State: Initialized, Role: FOLLOWER
I20260810 07:25:58.986073 28312 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd46c86fb5cc404383cfe9ccc3d1fcca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd46c86fb5cc404383cfe9ccc3d1fcca" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46283 } }
I20260810 07:25:58.986223 28312 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd46c86fb5cc404383cfe9ccc3d1fcca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:25:58.986306 28312 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd46c86fb5cc404383cfe9ccc3d1fcca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:25:58.986428 28312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd46c86fb5cc404383cfe9ccc3d1fcca [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:58.987318 28312 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd46c86fb5cc404383cfe9ccc3d1fcca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd46c86fb5cc404383cfe9ccc3d1fcca" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46283 } }
I20260810 07:25:58.987653 28312 leader_election.cc:302] T 00000000000000000000000000000000 P dd46c86fb5cc404383cfe9ccc3d1fcca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd46c86fb5cc404383cfe9ccc3d1fcca; no voters:
I20260810 07:25:58.987960 28312 leader_election.cc:288] T 00000000000000000000000000000000 P dd46c86fb5cc404383cfe9ccc3d1fcca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:25:58.988133 28317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd46c86fb5cc404383cfe9ccc3d1fcca [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:58.988464 28317 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd46c86fb5cc404383cfe9ccc3d1fcca [term 1 LEADER]: Becoming Leader. State: Replica: dd46c86fb5cc404383cfe9ccc3d1fcca, State: Running, Role: LEADER
I20260810 07:25:58.988767 28317 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd46c86fb5cc404383cfe9ccc3d1fcca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd46c86fb5cc404383cfe9ccc3d1fcca" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46283 } }
I20260810 07:25:58.989290 28312 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd46c86fb5cc404383cfe9ccc3d1fcca [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:25:58.990523 28319 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd46c86fb5cc404383cfe9ccc3d1fcca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd46c86fb5cc404383cfe9ccc3d1fcca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd46c86fb5cc404383cfe9ccc3d1fcca" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46283 } } }
I20260810 07:25:58.990672 28319 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd46c86fb5cc404383cfe9ccc3d1fcca [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:58.991034 28321 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd46c86fb5cc404383cfe9ccc3d1fcca [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd46c86fb5cc404383cfe9ccc3d1fcca. Latest consensus state: current_term: 1 leader_uuid: "dd46c86fb5cc404383cfe9ccc3d1fcca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd46c86fb5cc404383cfe9ccc3d1fcca" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46283 } } }
I20260810 07:25:58.991107 28333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:25:58.991658 28321 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd46c86fb5cc404383cfe9ccc3d1fcca [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:58.993873 28333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:25:58.999770 28333 catalog_manager.cc:1347] Generated new cluster ID: 8f9838786ee444a189eff98d66917288
I20260810 07:25:58.999846 28333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:25:59.008226 28333 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:25:59.009442 28333 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:25:59.018080 28333 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd46c86fb5cc404383cfe9ccc3d1fcca: Generated new TSK 0
I20260810 07:25:59.019275 28333 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:25:59.029004 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:0
--local_ip_for_outbound_sockets=127.12.157.193
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:46283
--builtin_ntp_servers=127.12.157.212:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 07:25:59.150193 28350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:59.150477 28350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:59.150518 28350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:59.154479 28350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:59.154543 28350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:59.154596 28350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:25:59.159400 28350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:46283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:59.160490 28350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:59.161901 28350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:59.172020 28363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:59.172008 28359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:59.172008 28357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:59.172274 28350 server_base.cc:1034] running on GCE node
I20260810 07:25:59.172775 28350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:59.173518 28350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:59.174739 28350 hybrid_clock.cc:648] HybridClock initialized: now 1786346759174726 us; error 33 us; skew 500 ppm
I20260810 07:25:59.176800 28350 webserver.cc:460] Webserver started at http://127.12.157.193:32893/ using document root <none> and password file <none>
I20260810 07:25:59.177517 28350 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:59.177615 28350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:59.177857 28350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:59.180030 28350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/data/instance:
uuid: "674e8924e49a4db0a6f4fc2624bfd97a"
format_stamp: "Formatted at 2026-08-10 07:25:59 on dist-test-slave-ffrd"
server_key: "264c9de4bafff37d0572258628517f9c"
server_key_iv: "dd037da56d193490d874f7787867cfaa"
server_key_version: "encryptionkey@0"
I20260810 07:25:59.180615 28350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance:
uuid: "674e8924e49a4db0a6f4fc2624bfd97a"
format_stamp: "Formatted at 2026-08-10 07:25:59 on dist-test-slave-ffrd"
server_key: "264c9de4bafff37d0572258628517f9c"
server_key_iv: "dd037da56d193490d874f7787867cfaa"
server_key_version: "encryptionkey@0"
I20260810 07:25:59.184688 28350 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 07:25:59.187469 28376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:59.188822 28350 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 07:25:59.188997 28350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "674e8924e49a4db0a6f4fc2624bfd97a"
format_stamp: "Formatted at 2026-08-10 07:25:59 on dist-test-slave-ffrd"
server_key: "264c9de4bafff37d0572258628517f9c"
server_key_iv: "dd037da56d193490d874f7787867cfaa"
server_key_version: "encryptionkey@0"
I20260810 07:25:59.189137 28350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:59.210368 28350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:59.211155 28350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:59.211378 28350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:59.212002 28350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:59.213048 28350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:59.213124 28350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:59.213197 28350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:59.213249 28350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:59.222463 28350 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:37985
I20260810 07:25:59.222527 28565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:37985 every 8 connection(s)
I20260810 07:25:59.223469 28350 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:25:59.228102 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 28350
I20260810 07:25:59.228205 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance
I20260810 07:25:59.228490 12919 external_mini_cluster.cc:1428] Setting key 0c66b7ce90d5d9572f580fac027b55b6
I20260810 07:25:59.230854 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:0
--local_ip_for_outbound_sockets=127.12.157.194
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:46283
--builtin_ntp_servers=127.12.157.212:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 07:25:59.234735 28315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:59.225690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:39323 (local address 127.12.157.254:46283)
0810 07:25:59.225854 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:59.225858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:59.227089 (+ 1231us) server_negotiation.cc:407] Connection header received
0810 07:25:59.228669 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:59.228673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:59.228735 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:59.228835 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:59.230077 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:59.230896 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:59.232237 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:59.232650 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:59.232786 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:59.234208 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:59.234211 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:59.234213 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:25:59.234227 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:59.234247 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:25:59.234255 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:59.234337 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:59.234515 (+ 178us) server_negotiation.cc:299] Negotiation successful
0810 07:25:59.234582 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 07:25:59.235963 28568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:59.226070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46283 (local address 127.12.157.193:39323)
0810 07:25:59.226904 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:59.226965 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:59.228414 (+ 1449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:59.229079 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:59.229089 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:59.229192 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:59.229893 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:59.229910 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:59.231087 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:59.231092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:59.232094 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:59.232105 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:59.232994 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:59.232999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:59.233172 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:59.233915 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:25:59.233917 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:59.233959 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:25:59.234030 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:59.234341 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:59.234363 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:59.234369 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 07:25:59.234518 (+ 149us) client_negotiation.cc:240] Negotiation successful
0810 07:25:59.234853 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":117,"threads_started":1}
I20260810 07:25:59.237500 28566 heartbeater.cc:347] Connected to a master server at 127.12.157.254:46283
I20260810 07:25:59.237949 28566 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:59.238656 28566 heartbeater.cc:511] Master 127.12.157.254:46283 requested a full tablet report, sending...
I20260810 07:25:59.240518 28259 ts_manager.cc:194] Registered new tserver with Master: 674e8924e49a4db0a6f4fc2624bfd97a (127.12.157.193:37985)
I20260810 07:25:59.241812 28259 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.157.193:39323
W20260810 07:25:59.349712 28569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:59.349975 28569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:59.350014 28569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:59.354027 28569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:59.354084 28569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:59.354166 28569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:25:59.358867 28569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:46283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:59.359987 28569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:59.361428 28569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:59.371239 28579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:59.371239 28575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:59.371430 28569 server_base.cc:1034] running on GCE node
W20260810 07:25:59.371251 28577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:59.371948 28569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:59.372606 28569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:59.373839 28569 hybrid_clock.cc:648] HybridClock initialized: now 1786346759373817 us; error 37 us; skew 500 ppm
I20260810 07:25:59.375923 28569 webserver.cc:460] Webserver started at http://127.12.157.194:39777/ using document root <none> and password file <none>
I20260810 07:25:59.376525 28569 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:59.376623 28569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:59.376875 28569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:59.379050 28569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/data/instance:
uuid: "baf79a40002b4fb498556fbf398de9f0"
format_stamp: "Formatted at 2026-08-10 07:25:59 on dist-test-slave-ffrd"
server_key: "4a530e764e24223407ee83e827d54717"
server_key_iv: "65d86929362a423ed10be0c08cff0fbc"
server_key_version: "encryptionkey@0"
I20260810 07:25:59.379758 28569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance:
uuid: "baf79a40002b4fb498556fbf398de9f0"
format_stamp: "Formatted at 2026-08-10 07:25:59 on dist-test-slave-ffrd"
server_key: "4a530e764e24223407ee83e827d54717"
server_key_iv: "65d86929362a423ed10be0c08cff0fbc"
server_key_version: "encryptionkey@0"
I20260810 07:25:59.383774 28569 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 07:25:59.387248 28586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:59.389153 28569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:25:59.389321 28569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "baf79a40002b4fb498556fbf398de9f0"
format_stamp: "Formatted at 2026-08-10 07:25:59 on dist-test-slave-ffrd"
server_key: "4a530e764e24223407ee83e827d54717"
server_key_iv: "65d86929362a423ed10be0c08cff0fbc"
server_key_version: "encryptionkey@0"
I20260810 07:25:59.389488 28569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:59.416667 28569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:59.417532 28569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:59.417750 28569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:59.418390 28569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:59.419394 28569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:59.419469 28569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:59.419546 28569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:59.419592 28569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:59.428903 28569 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:40813
I20260810 07:25:59.429059 28770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:40813 every 8 connection(s)
I20260810 07:25:59.429942 28569 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:25:59.439534 28315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:59.431891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:46211 (local address 127.12.157.254:46283)
0810 07:25:59.432086 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:59.432093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:59.433226 (+ 1133us) server_negotiation.cc:407] Connection header received
0810 07:25:59.434630 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:59.434634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:59.434694 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:59.434803 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:59.435693 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:59.436229 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:59.437491 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:59.437761 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:59.437904 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:59.439013 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:59.439017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:59.439019 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:25:59.439031 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:59.439050 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:25:59.439058 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:59.439118 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:59.439257 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 07:25:59.439307 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 07:25:59.439641 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 28569
I20260810 07:25:59.439754 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance
I20260810 07:25:59.440066 12919 external_mini_cluster.cc:1428] Setting key 6079245c640e081e2dc4a9c20dff6d3d
I20260810 07:25:59.440363 28773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:59.432326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46283 (local address 127.12.157.194:46211)
0810 07:25:59.433079 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:59.433121 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:59.434419 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:59.434987 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:59.434994 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:59.435075 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:59.435550 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:59.435562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:59.436370 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:59.436373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:59.437363 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:59.437375 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:59.437948 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:59.437952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:59.438121 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:59.438775 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:25:59.438777 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:59.438815 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:25:59.438876 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:59.439136 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:59.439143 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:59.439148 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:25:59.439241 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 07:25:59.439470 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":121,"threads_started":1}
I20260810 07:25:59.441689 28771 heartbeater.cc:347] Connected to a master server at 127.12.157.254:46283
I20260810 07:25:59.441947 28771 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:59.442250 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:0
--local_ip_for_outbound_sockets=127.12.157.195
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:46283
--builtin_ntp_servers=127.12.157.212:45033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 07:25:59.442524 28771 heartbeater.cc:511] Master 127.12.157.254:46283 requested a full tablet report, sending...
I20260810 07:25:59.443585 28259 ts_manager.cc:194] Registered new tserver with Master: baf79a40002b4fb498556fbf398de9f0 (127.12.157.194:40813)
I20260810 07:25:59.444363 28259 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.157.194:46211
W20260810 07:25:59.563552 28774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:59.563854 28774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:59.563894 28774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:59.568046 28774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:59.568109 28774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:59.568204 28774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:25:59.573239 28774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:45033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:46283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:59.574430 28774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:59.575891 28774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 07:25:59.586458 28774 server_base.cc:1034] running on GCE node
W20260810 07:25:59.586418 28785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:59.586413 28782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:59.586429 28783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:59.587098 28774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:59.587816 28774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:59.589072 28774 hybrid_clock.cc:648] HybridClock initialized: now 1786346759589032 us; error 52 us; skew 500 ppm
I20260810 07:25:59.591534 28774 webserver.cc:460] Webserver started at http://127.12.157.195:46535/ using document root <none> and password file <none>
I20260810 07:25:59.592154 28774 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:59.592252 28774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:59.592617 28774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:59.595062 28774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/data/instance:
uuid: "3f3ad99db71942fe8a355d45c9f09b70"
format_stamp: "Formatted at 2026-08-10 07:25:59 on dist-test-slave-ffrd"
server_key: "a97490e62c1d8346a448d7840a44d02a"
server_key_iv: "ff523b06228ea1423ebda4c98d55ea98"
server_key_version: "encryptionkey@0"
I20260810 07:25:59.595631 28774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance:
uuid: "3f3ad99db71942fe8a355d45c9f09b70"
format_stamp: "Formatted at 2026-08-10 07:25:59 on dist-test-slave-ffrd"
server_key: "a97490e62c1d8346a448d7840a44d02a"
server_key_iv: "ff523b06228ea1423ebda4c98d55ea98"
server_key_version: "encryptionkey@0"
I20260810 07:25:59.599372 28774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 07:25:59.601998 28794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:59.603168 28774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 07:25:59.603305 28774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "3f3ad99db71942fe8a355d45c9f09b70"
format_stamp: "Formatted at 2026-08-10 07:25:59 on dist-test-slave-ffrd"
server_key: "a97490e62c1d8346a448d7840a44d02a"
server_key_iv: "ff523b06228ea1423ebda4c98d55ea98"
server_key_version: "encryptionkey@0"
I20260810 07:25:59.603421 28774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:59.621182 28774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:59.622053 28774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:59.622275 28774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:59.622936 28774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:25:59.623919 28774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:25:59.623993 28774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:59.624068 28774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:25:59.624112 28774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:59.634438 28774 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:46505
I20260810 07:25:59.634466 28963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:46505 every 8 connection(s)
I20260810 07:25:59.635492 28774 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:25:59.640733 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 28774
I20260810 07:25:59.640859 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance
I20260810 07:25:59.641167 12919 external_mini_cluster.cc:1428] Setting key 835ebacc0637a96c8e62fdae206efa00
I20260810 07:25:59.645866 28315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:59.637628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:41333 (local address 127.12.157.254:46283)
0810 07:25:59.637793 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:59.637797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:59.638885 (+ 1088us) server_negotiation.cc:407] Connection header received
0810 07:25:59.640291 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:59.640294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:59.640353 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:59.640435 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:59.641781 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:59.642675 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:59.644040 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:25:59.644252 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:25:59.644342 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:59.645409 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:59.645412 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:59.645414 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:25:59.645428 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:59.645454 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:25:59.645462 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:59.645522 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:59.645694 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 07:25:59.645748 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 07:25:59.646678 28967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:59.637978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46283 (local address 127.12.157.195:41333)
0810 07:25:59.638695 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0810 07:25:59.638737 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 07:25:59.640077 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:25:59.640727 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:25:59.640737 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:25:59.640861 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:25:59.641595 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:59.641613 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:59.642834 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:59.642836 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:59.643917 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:25:59.643928 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:25:59.644398 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:25:59.644401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:25:59.644546 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:25:59.645157 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:25:59.645159 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:25:59.645198 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:25:59.645262 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:25:59.645539 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:25:59.645546 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:25:59.645550 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:25:59.645646 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 07:25:59.645886 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":122,"threads_started":1}
I20260810 07:25:59.647801 28964 heartbeater.cc:347] Connected to a master server at 127.12.157.254:46283
I20260810 07:25:59.648023 28964 heartbeater.cc:464] Registering TS with master...
I20260810 07:25:59.648499 28964 heartbeater.cc:511] Master 127.12.157.254:46283 requested a full tablet report, sending...
I20260810 07:25:59.649549 28259 ts_manager.cc:194] Registered new tserver with Master: 3f3ad99db71942fe8a355d45c9f09b70 (127.12.157.195:46505)
I20260810 07:25:59.650152 28259 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.157.195:41333
I20260810 07:25:59.655172 12919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 07:25:59.658174 28315 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:25:59.657225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48304 (local address 127.12.157.254:46283)
0810 07:25:59.657387 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:59.657392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:59.657567 (+ 175us) server_negotiation.cc:407] Connection header received
0810 07:25:59.657619 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:59.657622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:59.657666 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:59.657752 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:59.658017 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48304: BlockingRecv error: recv got EOF from 127.0.0.1:48304 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 07:25:59.658236 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 28350
I20260810 07:25:59.674780 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 28569
I20260810 07:25:59.692718 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 28774
I20260810 07:25:59.712556 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 28213
2026-08-10T07:25:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (973 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T07:25:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:25:59Z Disabled control of system clock
I20260810 07:25:59.748613 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:37845
--webserver_interface=127.12.157.254
--webserver_port=0
--builtin_ntp_servers=127.12.157.212:43069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:37845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 07:25:59.863580 28978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:25:59.863827 28978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:25:59.863865 28978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:25:59.867803 28978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:25:59.867856 28978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:25:59.867871 28978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:25:59.867888 28978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:25:59.867903 28978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:25:59.872918 28978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:43069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:37845
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:37845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:25:59.873997 28978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:25:59.875365 28978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:25:59.883769 28985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:59.883772 28984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:25:59.883819 28988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:25:59.884305 28978 server_base.cc:1034] running on GCE node
I20260810 07:25:59.884907 28978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:25:59.886072 28978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:25:59.887282 28978 hybrid_clock.cc:648] HybridClock initialized: now 1786346759887254 us; error 41 us; skew 500 ppm
I20260810 07:25:59.889490 28978 webserver.cc:460] Webserver started at http://127.12.157.254:46883/ using document root <none> and password file <none>
I20260810 07:25:59.890098 28978 fs_manager.cc:359] Metadata directory not provided
I20260810 07:25:59.890200 28978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:25:59.890445 28978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:25:59.892558 28978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "a0039d68f74e4d0e8d12d0b856fe2f65"
format_stamp: "Formatted at 2026-08-10 07:25:59 on dist-test-slave-ffrd"
server_key: "2202392cef1d18418c165d2657249da5"
server_key_iv: "39b3330808e876d17f0901ad95a2c405"
server_key_version: "encryptionkey@0"
I20260810 07:25:59.893209 28978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "a0039d68f74e4d0e8d12d0b856fe2f65"
format_stamp: "Formatted at 2026-08-10 07:25:59 on dist-test-slave-ffrd"
server_key: "2202392cef1d18418c165d2657249da5"
server_key_iv: "39b3330808e876d17f0901ad95a2c405"
server_key_version: "encryptionkey@0"
I20260810 07:25:59.896925 28978 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 07:25:59.899613 28996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:25:59.900923 28978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 07:25:59.901145 28978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "a0039d68f74e4d0e8d12d0b856fe2f65"
format_stamp: "Formatted at 2026-08-10 07:25:59 on dist-test-slave-ffrd"
server_key: "2202392cef1d18418c165d2657249da5"
server_key_iv: "39b3330808e876d17f0901ad95a2c405"
server_key_version: "encryptionkey@0"
I20260810 07:25:59.901281 28978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:25:59.906107 28978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:25:59.906734 28978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:25:59.906941 28978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:25:59.914192 28978 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:37845
I20260810 07:25:59.914197 29079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:37845 every 8 connection(s)
I20260810 07:25:59.915127 28978 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:25:59.918409 29080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:25:59.924165 29080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0039d68f74e4d0e8d12d0b856fe2f65: Bootstrap starting.
I20260810 07:25:59.925206 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 28978
I20260810 07:25:59.925355 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:25:59.925638 12919 external_mini_cluster.cc:1428] Setting key 08281306c537326ba63c770c7d0eb78f
I20260810 07:25:59.927172 29080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0039d68f74e4d0e8d12d0b856fe2f65: Neither blocks nor log segments found. Creating new log.
I20260810 07:25:59.928071 29080 log.cc:824] T 00000000000000000000000000000000 P a0039d68f74e4d0e8d12d0b856fe2f65: Log is configured to *not* fsync() on all Append() calls
I20260810 07:25:59.930418 29080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0039d68f74e4d0e8d12d0b856fe2f65: No bootstrap required, opened a new log
I20260810 07:25:59.931289 29083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:25:59.926827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36268 (local address 127.12.157.254:37845)
0810 07:25:59.927570 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0810 07:25:59.927588 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 07:25:59.927624 (+ 36us) server_negotiation.cc:407] Connection header received
0810 07:25:59.928355 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:25:59.928375 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:25:59.928762 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:25:59.929200 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:25:59.930027 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:25:59.930032 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:25:59.930043 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:25:59.930055 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:25:59.930113 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:25:59.930130 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:25:59.930262 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 07:25:59.930342 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 07:25:59.930606 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":146,"threads_started":1}
I20260810 07:25:59.934191 29080 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0039d68f74e4d0e8d12d0b856fe2f65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0039d68f74e4d0e8d12d0b856fe2f65" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37845 } }
I20260810 07:25:59.934419 29080 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0039d68f74e4d0e8d12d0b856fe2f65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:25:59.934480 29080 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0039d68f74e4d0e8d12d0b856fe2f65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0039d68f74e4d0e8d12d0b856fe2f65, State: Initialized, Role: FOLLOWER
I20260810 07:25:59.934881 29080 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0039d68f74e4d0e8d12d0b856fe2f65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0039d68f74e4d0e8d12d0b856fe2f65" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37845 } }
I20260810 07:25:59.935001 29080 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0039d68f74e4d0e8d12d0b856fe2f65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:25:59.935086 29080 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0039d68f74e4d0e8d12d0b856fe2f65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:25:59.935187 29080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0039d68f74e4d0e8d12d0b856fe2f65 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:25:59.936134 29080 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0039d68f74e4d0e8d12d0b856fe2f65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0039d68f74e4d0e8d12d0b856fe2f65" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37845 } }
I20260810 07:25:59.936480 29080 leader_election.cc:302] T 00000000000000000000000000000000 P a0039d68f74e4d0e8d12d0b856fe2f65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0039d68f74e4d0e8d12d0b856fe2f65; no voters:
I20260810 07:25:59.936813 29080 leader_election.cc:288] T 00000000000000000000000000000000 P a0039d68f74e4d0e8d12d0b856fe2f65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:25:59.937012 29086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0039d68f74e4d0e8d12d0b856fe2f65 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:25:59.937260 29086 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0039d68f74e4d0e8d12d0b856fe2f65 [term 1 LEADER]: Becoming Leader. State: Replica: a0039d68f74e4d0e8d12d0b856fe2f65, State: Running, Role: LEADER
I20260810 07:25:59.937690 29086 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0039d68f74e4d0e8d12d0b856fe2f65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0039d68f74e4d0e8d12d0b856fe2f65" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37845 } }
I20260810 07:25:59.938169 29080 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0039d68f74e4d0e8d12d0b856fe2f65 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:25:59.939710 29087 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0039d68f74e4d0e8d12d0b856fe2f65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0039d68f74e4d0e8d12d0b856fe2f65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0039d68f74e4d0e8d12d0b856fe2f65" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37845 } } }
I20260810 07:25:59.939743 29091 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0039d68f74e4d0e8d12d0b856fe2f65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0039d68f74e4d0e8d12d0b856fe2f65. Latest consensus state: current_term: 1 leader_uuid: "a0039d68f74e4d0e8d12d0b856fe2f65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0039d68f74e4d0e8d12d0b856fe2f65" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37845 } } }
I20260810 07:25:59.939855 29087 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0039d68f74e4d0e8d12d0b856fe2f65 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:59.939855 29091 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0039d68f74e4d0e8d12d0b856fe2f65 [sys.catalog]: This master's current role is: LEADER
I20260810 07:25:59.940282 29109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:25:59.943713 29109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:25:59.949787 29109 catalog_manager.cc:1347] Generated new cluster ID: 16547bd3b2eb4b8b9323b424c19dbe62
I20260810 07:25:59.949870 29109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:25:59.963982 29109 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:25:59.964339 29109 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:25:59.978642 29109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0039d68f74e4d0e8d12d0b856fe2f65: Generated new TSK 0
I20260810 07:25:59.979421 29109 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:25:59.989603 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:0
--local_ip_for_outbound_sockets=127.12.157.193
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37845
--builtin_ntp_servers=127.12.157.212:43069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 07:26:00.115967 29118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:26:00.116323 29118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:26:00.116369 29118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:26:00.120496 29118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:26:00.120565 29118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:26:00.120623 29118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:26:00.125510 29118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:43069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:26:00.126739 29118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:26:00.128187 29118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:26:00.138370 29124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:26:00.138362 29128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:26:00.138577 29125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:26:00.138787 29118 server_base.cc:1034] running on GCE node
I20260810 07:26:00.139312 29118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:26:00.140056 29118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:26:00.141254 29118 hybrid_clock.cc:648] HybridClock initialized: now 1786346760141244 us; error 46 us; skew 500 ppm
I20260810 07:26:00.143250 29118 webserver.cc:460] Webserver started at http://127.12.157.193:41751/ using document root <none> and password file <none>
I20260810 07:26:00.143918 29118 fs_manager.cc:359] Metadata directory not provided
I20260810 07:26:00.144019 29118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:26:00.144266 29118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:26:00.146540 29118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/data/instance:
uuid: "f2886b752cff47bcb9dac7ed55bc4866"
format_stamp: "Formatted at 2026-08-10 07:26:00 on dist-test-slave-ffrd"
server_key: "8da59ff71455fcf674d48da686ba3389"
server_key_iv: "aad4a1bf21b4f2e590ed2b8ac332ece1"
server_key_version: "encryptionkey@0"
I20260810 07:26:00.147149 29118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance:
uuid: "f2886b752cff47bcb9dac7ed55bc4866"
format_stamp: "Formatted at 2026-08-10 07:26:00 on dist-test-slave-ffrd"
server_key: "8da59ff71455fcf674d48da686ba3389"
server_key_iv: "aad4a1bf21b4f2e590ed2b8ac332ece1"
server_key_version: "encryptionkey@0"
I20260810 07:26:00.151034 29118 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 07:26:00.154384 29138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:00.155652 29118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 07:26:00.155800 29118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "f2886b752cff47bcb9dac7ed55bc4866"
format_stamp: "Formatted at 2026-08-10 07:26:00 on dist-test-slave-ffrd"
server_key: "8da59ff71455fcf674d48da686ba3389"
server_key_iv: "aad4a1bf21b4f2e590ed2b8ac332ece1"
server_key_version: "encryptionkey@0"
I20260810 07:26:00.155927 29118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:26:00.160827 29118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:26:00.161490 29118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:26:00.161684 29118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:26:00.162248 29118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:26:00.163182 29118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:26:00.163254 29118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:00.163323 29118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:26:00.163373 29118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:00.172343 29118 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:38303
I20260810 07:26:00.172374 29299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:38303 every 8 connection(s)
I20260810 07:26:00.173327 29118 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:26:00.177359 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 29118
I20260810 07:26:00.177527 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance
I20260810 07:26:00.177851 12919 external_mini_cluster.cc:1428] Setting key a78fb5dd3e7fd6dc5efea78cac9019a3
I20260810 07:26:00.179999 29083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:00.175272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:36653 (local address 127.12.157.254:37845)
0810 07:26:00.175450 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:00.175455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:00.176467 (+ 1012us) server_negotiation.cc:407] Connection header received
0810 07:26:00.178080 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:00.178083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:00.178124 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:00.178204 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:26:00.179497 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:26:00.179501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:26:00.179504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:26:00.179517 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:26:00.179543 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:26:00.179553 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:26:00.179635 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:00.179850 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 07:26:00.179896 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 07:26:00.180176 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:0
--local_ip_for_outbound_sockets=127.12.157.194
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37845
--builtin_ntp_servers=127.12.157.212:43069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 07:26:00.181170 29302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:00.175668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37845 (local address 127.12.157.193:36653)
0810 07:26:00.176325 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:00.176363 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 07:26:00.177836 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:26:00.178394 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:26:00.178402 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:26:00.178519 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:26:00.179176 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:26:00.179178 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:26:00.179237 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:26:00.179329 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:26:00.179654 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:26:00.179664 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:26:00.179670 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 07:26:00.179849 (+ 179us) client_negotiation.cc:240] Negotiation successful
0810 07:26:00.180107 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":128,"threads_started":1}
I20260810 07:26:00.183068 29300 heartbeater.cc:347] Connected to a master server at 127.12.157.254:37845
I20260810 07:26:00.183318 29300 heartbeater.cc:464] Registering TS with master...
I20260810 07:26:00.183755 29300 heartbeater.cc:511] Master 127.12.157.254:37845 requested a full tablet report, sending...
I20260810 07:26:00.185488 29019 ts_manager.cc:194] Registered new tserver with Master: f2886b752cff47bcb9dac7ed55bc4866 (127.12.157.193:38303)
W20260810 07:26:00.300191 29303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:26:00.300484 29303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:26:00.300556 29303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:26:00.304569 29303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:26:00.304642 29303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:26:00.304726 29303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:26:00.309752 29303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:43069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:26:00.310797 29303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:26:00.312235 29303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:26:00.321923 29309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:26:00.322031 29313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:26:00.321924 29310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:26:00.322268 29303 server_base.cc:1034] running on GCE node
I20260810 07:26:00.322669 29303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:26:00.323352 29303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:26:00.324566 29303 hybrid_clock.cc:648] HybridClock initialized: now 1786346760324544 us; error 31 us; skew 500 ppm
I20260810 07:26:00.326624 29303 webserver.cc:460] Webserver started at http://127.12.157.194:38793/ using document root <none> and password file <none>
I20260810 07:26:00.327222 29303 fs_manager.cc:359] Metadata directory not provided
I20260810 07:26:00.327329 29303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:26:00.327567 29303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:26:00.329800 29303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/data/instance:
uuid: "b0ff9b2efa4e4797b6f183c68e27c773"
format_stamp: "Formatted at 2026-08-10 07:26:00 on dist-test-slave-ffrd"
server_key: "de49b61d7e6bb82dbe6566052c67e661"
server_key_iv: "b71436f33a31e3f4f3729642422a398d"
server_key_version: "encryptionkey@0"
I20260810 07:26:00.330382 29303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance:
uuid: "b0ff9b2efa4e4797b6f183c68e27c773"
format_stamp: "Formatted at 2026-08-10 07:26:00 on dist-test-slave-ffrd"
server_key: "de49b61d7e6bb82dbe6566052c67e661"
server_key_iv: "b71436f33a31e3f4f3729642422a398d"
server_key_version: "encryptionkey@0"
I20260810 07:26:00.334161 29303 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 07:26:00.336915 29320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:00.338495 29303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:26:00.338667 29303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "b0ff9b2efa4e4797b6f183c68e27c773"
format_stamp: "Formatted at 2026-08-10 07:26:00 on dist-test-slave-ffrd"
server_key: "de49b61d7e6bb82dbe6566052c67e661"
server_key_iv: "b71436f33a31e3f4f3729642422a398d"
server_key_version: "encryptionkey@0"
I20260810 07:26:00.338822 29303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:26:00.343734 29303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:26:00.344348 29303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:26:00.344532 29303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:26:00.345134 29303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:26:00.346042 29303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:26:00.346092 29303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:00.346131 29303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:26:00.346186 29303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:00.355309 29303 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:45651
I20260810 07:26:00.355350 29486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:45651 every 8 connection(s)
I20260810 07:26:00.356209 29303 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:26:00.359320 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 29303
I20260810 07:26:00.359459 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance
I20260810 07:26:00.359760 12919 external_mini_cluster.cc:1428] Setting key f4639c3754419207944f4c2f064dcc4b
I20260810 07:26:00.361933 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:0
--local_ip_for_outbound_sockets=127.12.157.195
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37845
--builtin_ntp_servers=127.12.157.212:43069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 07:26:00.362980 29083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:00.358123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:48949 (local address 127.12.157.254:37845)
0810 07:26:00.358266 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:00.358270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:00.359460 (+ 1190us) server_negotiation.cc:407] Connection header received
0810 07:26:00.361090 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:00.361093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:00.361149 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:00.361245 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:26:00.362542 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:26:00.362545 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:26:00.362547 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:26:00.362559 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:26:00.362578 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:26:00.362587 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:26:00.362659 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:00.362838 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 07:26:00.362877 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 07:26:00.364099 29492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:00.358511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37845 (local address 127.12.157.194:48949)
0810 07:26:00.359264 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:00.359323 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 07:26:00.360805 (+ 1482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:26:00.361430 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:26:00.361439 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:26:00.361547 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:26:00.362229 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:26:00.362234 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:26:00.362287 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:26:00.362384 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:26:00.362679 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:26:00.362691 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:26:00.362700 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 07:26:00.362878 (+ 178us) client_negotiation.cc:240] Negotiation successful
0810 07:26:00.363118 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":109,"threads_started":1}
I20260810 07:26:00.365254 29487 heartbeater.cc:347] Connected to a master server at 127.12.157.254:37845
I20260810 07:26:00.365501 29487 heartbeater.cc:464] Registering TS with master...
I20260810 07:26:00.365979 29487 heartbeater.cc:511] Master 127.12.157.254:37845 requested a full tablet report, sending...
I20260810 07:26:00.367182 29019 ts_manager.cc:194] Registered new tserver with Master: b0ff9b2efa4e4797b6f183c68e27c773 (127.12.157.194:45651)
W20260810 07:26:00.481158 29495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:26:00.481601 29495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:26:00.481737 29495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:26:00.485954 29495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:26:00.486007 29495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:26:00.486063 29495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:26:00.490868 29495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:43069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:37845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:26:00.491976 29495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:26:00.493463 29495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:26:00.503245 29501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:26:00.503247 29500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:26:00.503350 29504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:26:00.503526 29495 server_base.cc:1034] running on GCE node
I20260810 07:26:00.504043 29495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:26:00.504724 29495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:26:00.505911 29495 hybrid_clock.cc:648] HybridClock initialized: now 1786346760505882 us; error 51 us; skew 500 ppm
I20260810 07:26:00.507933 29495 webserver.cc:460] Webserver started at http://127.12.157.195:36137/ using document root <none> and password file <none>
I20260810 07:26:00.508502 29495 fs_manager.cc:359] Metadata directory not provided
I20260810 07:26:00.508561 29495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:26:00.508728 29495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:26:00.510891 29495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/data/instance:
uuid: "530c1a84f1bf4aa5ba9ce186ee9bced5"
format_stamp: "Formatted at 2026-08-10 07:26:00 on dist-test-slave-ffrd"
server_key: "8e048808356470d46686ed94150c38ef"
server_key_iv: "788368a0799950eeaf535d4c932dc863"
server_key_version: "encryptionkey@0"
I20260810 07:26:00.511397 29495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance:
uuid: "530c1a84f1bf4aa5ba9ce186ee9bced5"
format_stamp: "Formatted at 2026-08-10 07:26:00 on dist-test-slave-ffrd"
server_key: "8e048808356470d46686ed94150c38ef"
server_key_iv: "788368a0799950eeaf535d4c932dc863"
server_key_version: "encryptionkey@0"
I20260810 07:26:00.515225 29495 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 07:26:00.517807 29515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:00.519116 29495 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:26:00.519253 29495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "530c1a84f1bf4aa5ba9ce186ee9bced5"
format_stamp: "Formatted at 2026-08-10 07:26:00 on dist-test-slave-ffrd"
server_key: "8e048808356470d46686ed94150c38ef"
server_key_iv: "788368a0799950eeaf535d4c932dc863"
server_key_version: "encryptionkey@0"
I20260810 07:26:00.519367 29495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:26:00.523836 29495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:26:00.524434 29495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:26:00.524610 29495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:26:00.525220 29495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:26:00.526117 29495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:26:00.526166 29495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:00.526229 29495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:26:00.526262 29495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:00.535125 29495 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:34283
I20260810 07:26:00.535202 29670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:34283 every 8 connection(s)
I20260810 07:26:00.536059 29495 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:26:00.540164 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 29495
I20260810 07:26:00.540266 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance
I20260810 07:26:00.540570 12919 external_mini_cluster.cc:1428] Setting key a42ea2221f4e5afe4cacc7be3f2612c5
I20260810 07:26:00.543241 29083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:00.538054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:40401 (local address 127.12.157.254:37845)
0810 07:26:00.538201 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:00.538205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:00.539442 (+ 1237us) server_negotiation.cc:407] Connection header received
0810 07:26:00.541049 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:00.541056 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:00.541106 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:00.541205 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:26:00.542626 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:26:00.542630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:26:00.542633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:26:00.542646 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:26:00.542668 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:26:00.542678 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:26:00.542835 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:00.543018 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 07:26:00.543059 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 07:26:00.544168 29673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:00.538466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37845 (local address 127.12.157.195:40401)
0810 07:26:00.539298 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:00.539342 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 07:26:00.540756 (+ 1414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:26:00.541429 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:26:00.541438 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:26:00.541554 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:26:00.542360 (+ 806us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:26:00.542361 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:26:00.542403 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:26:00.542474 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:26:00.542832 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:26:00.542841 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:26:00.542846 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:26:00.543079 (+ 233us) client_negotiation.cc:240] Negotiation successful
0810 07:26:00.543334 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":173,"threads_started":1}
I20260810 07:26:00.545502 29671 heartbeater.cc:347] Connected to a master server at 127.12.157.254:37845
I20260810 07:26:00.545789 29671 heartbeater.cc:464] Registering TS with master...
I20260810 07:26:00.546259 29671 heartbeater.cc:511] Master 127.12.157.254:37845 requested a full tablet report, sending...
I20260810 07:26:00.547281 29019 ts_manager.cc:194] Registered new tserver with Master: 530c1a84f1bf4aa5ba9ce186ee9bced5 (127.12.157.195:34283)
I20260810 07:26:00.554749 12919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 07:26:00.557968 29083 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:26:00.556928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36272 (local address 127.12.157.254:37845)
0810 07:26:00.557108 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:00.557111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:00.557351 (+ 240us) server_negotiation.cc:407] Connection header received
0810 07:26:00.557441 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:00.557448 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:00.557494 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:00.557572 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:26:00.557827 (+ 255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36272: BlockingRecv error: recv got EOF from 127.0.0.1:36272 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260810 07:26:00.558491 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 29118
I20260810 07:26:00.573385 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 29303
I20260810 07:26:00.588814 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 29495
I20260810 07:26:00.603085 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 28978
2026-08-10T07:26:00Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (894 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:26:00 dist-test-slave-ffrd krb5kdc[29685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:26:00 dist-test-slave-ffrd krb5kdc[29685](info): set up 2 sockets
Aug 10 07:26:00 dist-test-slave-ffrd krb5kdc[29685](info): commencing operation
krb5kdc: starting...
W20260810 07:26:01.902045 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.269s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:26:01 dist-test-slave-ffrd krb5kdc[29685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:26:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:26:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:26:02.036420 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:40969
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:40969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:26:02.155326 29702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:26:02.155594 29702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:26:02.155634 29702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:26:02.159686 29702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:26:02.159749 29702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:26:02.159770 29702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:26:02.159787 29702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:26:02.159828 29702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:26:02.165081 29702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:40969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:40969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:26:02.166121 29702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:26:02.167147 29702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:26:02.175616 29711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:26:02.175621 29714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:26:02.175647 29710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:26:02.175980 29702 server_base.cc:1034] running on GCE node
I20260810 07:26:02.176507 29702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:26:02.177623 29702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:26:02.178814 29702 hybrid_clock.cc:648] HybridClock initialized: now 1786346762178795 us; error 31 us; skew 500 ppm
Aug 10 07:26:02 dist-test-slave-ffrd krb5kdc[29685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346762, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:26:02.181633 29702 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:26:02.182530 29702 webserver.cc:460] Webserver started at http://127.12.157.254:43919/ using document root <none> and password file <none>
I20260810 07:26:02.183081 29702 fs_manager.cc:359] Metadata directory not provided
I20260810 07:26:02.183173 29702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:26:02.183413 29702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:26:02.185544 29702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "a45081e31b534a74944723e890924707"
format_stamp: "Formatted at 2026-08-10 07:26:02 on dist-test-slave-ffrd"
server_key: "9f5e3f436220de61c57516f605051081"
server_key_iv: "fd305bbd684752fd974393a2f4708435"
server_key_version: "encryptionkey@0"
I20260810 07:26:02.186107 29702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "a45081e31b534a74944723e890924707"
format_stamp: "Formatted at 2026-08-10 07:26:02 on dist-test-slave-ffrd"
server_key: "9f5e3f436220de61c57516f605051081"
server_key_iv: "fd305bbd684752fd974393a2f4708435"
server_key_version: "encryptionkey@0"
I20260810 07:26:02.189848 29702 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:26:02.192266 29724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:02.193521 29702 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:26:02.193661 29702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "a45081e31b534a74944723e890924707"
format_stamp: "Formatted at 2026-08-10 07:26:02 on dist-test-slave-ffrd"
server_key: "9f5e3f436220de61c57516f605051081"
server_key_iv: "fd305bbd684752fd974393a2f4708435"
server_key_version: "encryptionkey@0"
I20260810 07:26:02.193774 29702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:26:02.205457 29702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:26:02.208807 29702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:26:02.209079 29702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:26:02.216629 29702 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:40969
I20260810 07:26:02.216989 29802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:40969 every 8 connection(s)
I20260810 07:26:02.217753 29702 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:26:02.220448 29804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:26:02.224511 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 29702
I20260810 07:26:02.224648 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:26:02.224931 12919 external_mini_cluster.cc:1428] Setting key b5741569480af44bef5f3cdc2f2f3aab
I20260810 07:26:02.226449 29804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a45081e31b534a74944723e890924707: Bootstrap starting.
I20260810 07:26:02.228868 29804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a45081e31b534a74944723e890924707: Neither blocks nor log segments found. Creating new log.
I20260810 07:26:02.229658 29804 log.cc:824] T 00000000000000000000000000000000 P a45081e31b534a74944723e890924707: Log is configured to *not* fsync() on all Append() calls
I20260810 07:26:02.231570 29804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a45081e31b534a74944723e890924707: No bootstrap required, opened a new log
Aug 10 07:26:02 dist-test-slave-ffrd krb5kdc[29685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:26:02.235644 29804 raft_consensus.cc:348] T 00000000000000000000000000000000 P a45081e31b534a74944723e890924707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a45081e31b534a74944723e890924707" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 40969 } }
I20260810 07:26:02.235904 29804 raft_consensus.cc:374] T 00000000000000000000000000000000 P a45081e31b534a74944723e890924707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:26:02.235991 29804 raft_consensus.cc:729] T 00000000000000000000000000000000 P a45081e31b534a74944723e890924707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a45081e31b534a74944723e890924707, State: Initialized, Role: FOLLOWER
I20260810 07:26:02.236511 29804 consensus_queue.cc:260] T 00000000000000000000000000000000 P a45081e31b534a74944723e890924707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a45081e31b534a74944723e890924707" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 40969 } }
I20260810 07:26:02.236678 29804 raft_consensus.cc:388] T 00000000000000000000000000000000 P a45081e31b534a74944723e890924707 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:26:02.236750 29804 raft_consensus.cc:482] T 00000000000000000000000000000000 P a45081e31b534a74944723e890924707 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:26:02.236871 29804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a45081e31b534a74944723e890924707 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:26:02.238039 29804 raft_consensus.cc:504] T 00000000000000000000000000000000 P a45081e31b534a74944723e890924707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a45081e31b534a74944723e890924707" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 40969 } }
I20260810 07:26:02.238394 29804 leader_election.cc:302] T 00000000000000000000000000000000 P a45081e31b534a74944723e890924707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a45081e31b534a74944723e890924707; no voters:
I20260810 07:26:02.238775 29804 leader_election.cc:288] T 00000000000000000000000000000000 P a45081e31b534a74944723e890924707 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:26:02.238889 29812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a45081e31b534a74944723e890924707 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:26:02.239176 29812 raft_consensus.cc:686] T 00000000000000000000000000000000 P a45081e31b534a74944723e890924707 [term 1 LEADER]: Becoming Leader. State: Replica: a45081e31b534a74944723e890924707, State: Running, Role: LEADER
I20260810 07:26:02.239518 29812 consensus_queue.cc:237] T 00000000000000000000000000000000 P a45081e31b534a74944723e890924707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a45081e31b534a74944723e890924707" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 40969 } }
I20260810 07:26:02.240134 29804 sys_catalog.cc:564] T 00000000000000000000000000000000 P a45081e31b534a74944723e890924707 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:26:02.240525 29809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:02.226391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32838 (local address 127.12.157.254:40969)
0810 07:26:02.226836 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:02.226862 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:02.226899 (+ 37us) server_negotiation.cc:407] Connection header received
0810 07:26:02.227545 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:02.227561 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:02.227875 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:02.228144 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:26:02.229144 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:02.230220 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:02.231106 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:02.231579 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:02.231788 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:02.234811 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:26:02.234816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:26:02.234831 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:26:02.234861 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:26:02.237038 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:02.237476 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:02.237482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:02.237488 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:02.237630 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:02.237943 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:02.237946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:02.237948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:02.238501 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:26:02.238753 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:02.239110 (+ 357us) server_negotiation.cc:299] Negotiation successful
0810 07:26:02.239578 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":185,"threads_started":1}
I20260810 07:26:02.240688 29812 sys_catalog.cc:455] T 00000000000000000000000000000000 P a45081e31b534a74944723e890924707 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a45081e31b534a74944723e890924707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a45081e31b534a74944723e890924707" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 40969 } } }
I20260810 07:26:02.240840 29812 sys_catalog.cc:458] T 00000000000000000000000000000000 P a45081e31b534a74944723e890924707 [sys.catalog]: This master's current role is: LEADER
I20260810 07:26:02.241176 29812 sys_catalog.cc:455] T 00000000000000000000000000000000 P a45081e31b534a74944723e890924707 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a45081e31b534a74944723e890924707. Latest consensus state: current_term: 1 leader_uuid: "a45081e31b534a74944723e890924707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a45081e31b534a74944723e890924707" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 40969 } } }
I20260810 07:26:02.241271 29812 sys_catalog.cc:458] T 00000000000000000000000000000000 P a45081e31b534a74944723e890924707 [sys.catalog]: This master's current role is: LEADER
I20260810 07:26:02.241890 29821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:26:02.245448 29821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:26:02.251875 29821 catalog_manager.cc:1347] Generated new cluster ID: d707511969b24ea58543069892771843
I20260810 07:26:02.251972 29821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:26:02.269536 29821 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:26:02.270884 29821 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:26:02.277019 29821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a45081e31b534a74944723e890924707: Generated new TSK 0
I20260810 07:26:02.277819 29821 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
I20260810 07:26:02.315871 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:0
--local_ip_for_outbound_sockets=127.12.157.193
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:40969
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:26:02.432001 29844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:26:02.432282 29844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:26:02.432376 29844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:26:02.436990 29844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:26:02.437098 29844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:26:02.437229 29844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:26:02.443910 29844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:40969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:26:02.445398 29844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:26:02.446532 29844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:26:02.456663 29852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:26:02.456669 29851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:26:02.456642 29854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:26:02.456861 29844 server_base.cc:1034] running on GCE node
I20260810 07:26:02.457333 29844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:26:02.458043 29844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:26:02.459254 29844 hybrid_clock.cc:648] HybridClock initialized: now 1786346762459222 us; error 46 us; skew 500 ppm
Aug 10 07:26:02 dist-test-slave-ffrd krb5kdc[29685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346762, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:26:02.466679 29844 init.cc:377] Logged in from keytab as kudu/127.12.157.193@KRBTEST.COM (short username kudu)
I20260810 07:26:02.467662 29844 webserver.cc:460] Webserver started at http://127.12.157.193:35027/ using document root <none> and password file <none>
I20260810 07:26:02.468201 29844 fs_manager.cc:359] Metadata directory not provided
I20260810 07:26:02.468262 29844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:26:02.468443 29844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:26:02.470534 29844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/data/instance:
uuid: "65b5fd650d1c46d5b985968e1ca2eba9"
format_stamp: "Formatted at 2026-08-10 07:26:02 on dist-test-slave-ffrd"
server_key: "ac8ef47a4ef8c6b841541d8194f06139"
server_key_iv: "344b04441d1d8ef4c569460dcd2f28d0"
server_key_version: "encryptionkey@0"
I20260810 07:26:02.471055 29844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance:
uuid: "65b5fd650d1c46d5b985968e1ca2eba9"
format_stamp: "Formatted at 2026-08-10 07:26:02 on dist-test-slave-ffrd"
server_key: "ac8ef47a4ef8c6b841541d8194f06139"
server_key_iv: "344b04441d1d8ef4c569460dcd2f28d0"
server_key_version: "encryptionkey@0"
I20260810 07:26:02.474942 29844 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260810 07:26:02.477411 29861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:02.478726 29844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:26:02.478828 29844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "65b5fd650d1c46d5b985968e1ca2eba9"
format_stamp: "Formatted at 2026-08-10 07:26:02 on dist-test-slave-ffrd"
server_key: "ac8ef47a4ef8c6b841541d8194f06139"
server_key_iv: "344b04441d1d8ef4c569460dcd2f28d0"
server_key_version: "encryptionkey@0"
I20260810 07:26:02.478912 29844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:26:02.500766 29844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:26:02.503756 29844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:26:02.503916 29844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:26:02.504452 29844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:26:02.505244 29844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:26:02.505300 29844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:02.505342 29844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:26:02.505364 29844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:02.514451 29844 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:46429
I20260810 07:26:02.514483 30012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:46429 every 8 connection(s)
I20260810 07:26:02.515331 29844 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:26:02.524014 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 29844
I20260810 07:26:02.524156 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance
I20260810 07:26:02.524482 12919 external_mini_cluster.cc:1428] Setting key 86a4de5064d2ec926b7e37abbeda4b13
Aug 10 07:26:02 dist-test-slave-ffrd krb5kdc[29685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346762, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:26:02.529600 29809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:02.517199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:39751 (local address 127.12.157.254:40969)
0810 07:26:02.517346 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:02.517349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:02.518216 (+ 867us) server_negotiation.cc:407] Connection header received
0810 07:26:02.519657 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:02.519661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:02.519724 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:02.519811 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:26:02.521015 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:02.521579 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:02.522719 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:02.522953 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:02.523068 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:02.525931 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:26:02.525938 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:26:02.525945 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:26:02.525977 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:26:02.527487 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:02.528015 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:02.528023 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:02.528024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:02.528098 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:02.528443 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:02.528446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:02.528448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:02.528622 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:26:02.528730 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:02.529302 (+ 572us) server_negotiation.cc:299] Negotiation successful
0810 07:26:02.529422 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 07:26:02.530361 30015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:02.517567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:40969 (local address 127.12.157.193:39751)
0810 07:26:02.518058 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:02.518101 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 07:26:02.519439 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:26:02.519988 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:26:02.519995 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:26:02.520367 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:26:02.520843 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:02.520858 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:02.521714 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:02.521717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:02.522585 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:02.522595 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:02.523111 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:02.523115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:02.523283 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:02.523885 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:26:02.523909 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:26:02.525711 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:26:02.527614 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:26:02.527620 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:26:02.527633 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:26:02.527902 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:26:02.528192 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:26:02.528196 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:26:02.528198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:26:02.528339 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:26:02.528731 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:26:02.528739 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:26:02.529057 (+ 318us) client_negotiation.cc:769] Sending connection context
0810 07:26:02.529235 (+ 178us) client_negotiation.cc:240] Negotiation successful
0810 07:26:02.529564 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":121,"threads_started":1}
I20260810 07:26:02.531595 30013 heartbeater.cc:347] Connected to a master server at 127.12.157.254:40969
I20260810 07:26:02.531859 30013 heartbeater.cc:464] Registering TS with master...
I20260810 07:26:02.532349 30013 heartbeater.cc:511] Master 127.12.157.254:40969 requested a full tablet report, sending...
I20260810 07:26:02.534075 29749 ts_manager.cc:194] Registered new tserver with Master: 65b5fd650d1c46d5b985968e1ca2eba9 (127.12.157.193:46429)
WARNING: no policy specified for kudu/127.12.157.194@KRBTEST.COM; defaulting to no policy
I20260810 07:26:02.535766 29749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.193@KRBTEST.COM'} at 127.12.157.193:39751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
I20260810 07:26:02.553929 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:0
--local_ip_for_outbound_sockets=127.12.157.194
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:40969
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:26:02.671715 30020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:26:02.671989 30020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:26:02.672034 30020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:26:02.676159 30020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:26:02.676222 30020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:26:02.676330 30020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:26:02.681643 30020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:40969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:26:02.682737 30020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:26:02.683799 30020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:26:02.693611 30028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:26:02.693626 30031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:26:02.693626 30029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:26:02.694090 30020 server_base.cc:1034] running on GCE node
I20260810 07:26:02.694515 30020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:26:02.695153 30020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:26:02.696352 30020 hybrid_clock.cc:648] HybridClock initialized: now 1786346762696331 us; error 30 us; skew 500 ppm
Aug 10 07:26:02 dist-test-slave-ffrd krb5kdc[29685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346762, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:26:02.699282 30020 init.cc:377] Logged in from keytab as kudu/127.12.157.194@KRBTEST.COM (short username kudu)
I20260810 07:26:02.700235 30020 webserver.cc:460] Webserver started at http://127.12.157.194:41703/ using document root <none> and password file <none>
I20260810 07:26:02.700814 30020 fs_manager.cc:359] Metadata directory not provided
I20260810 07:26:02.700907 30020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:26:02.701169 30020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:26:02.703214 30020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/data/instance:
uuid: "6bafaee674a0461c9f5bff5c7f4c6820"
format_stamp: "Formatted at 2026-08-10 07:26:02 on dist-test-slave-ffrd"
server_key: "f1c56a1951118ac6daf6132430321f45"
server_key_iv: "198d4925c574dd3aae25f49f82160e6d"
server_key_version: "encryptionkey@0"
I20260810 07:26:02.703800 30020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance:
uuid: "6bafaee674a0461c9f5bff5c7f4c6820"
format_stamp: "Formatted at 2026-08-10 07:26:02 on dist-test-slave-ffrd"
server_key: "f1c56a1951118ac6daf6132430321f45"
server_key_iv: "198d4925c574dd3aae25f49f82160e6d"
server_key_version: "encryptionkey@0"
I20260810 07:26:02.707696 30020 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 07:26:02.710376 30039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:02.711562 30020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 07:26:02.711699 30020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "6bafaee674a0461c9f5bff5c7f4c6820"
format_stamp: "Formatted at 2026-08-10 07:26:02 on dist-test-slave-ffrd"
server_key: "f1c56a1951118ac6daf6132430321f45"
server_key_iv: "198d4925c574dd3aae25f49f82160e6d"
server_key_version: "encryptionkey@0"
I20260810 07:26:02.711813 30020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:26:02.729902 30020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:26:02.733366 30020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:26:02.733578 30020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:26:02.734196 30020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:26:02.735049 30020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:26:02.735121 30020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:02.735196 30020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:26:02.735239 30020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:02.745012 30020 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:33123
I20260810 07:26:02.745060 30200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:33123 every 8 connection(s)
I20260810 07:26:02.745939 30020 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:26:02.751303 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 30020
I20260810 07:26:02.751433 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance
I20260810 07:26:02.751703 12919 external_mini_cluster.cc:1428] Setting key dbef40337b3ba0ecf0dc390e1a18356f
Aug 10 07:26:02 dist-test-slave-ffrd krb5kdc[29685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346762, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:26:02.760608 29809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:02.747703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:35745 (local address 127.12.157.254:40969)
0810 07:26:02.747840 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:02.747844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:02.748660 (+ 816us) server_negotiation.cc:407] Connection header received
0810 07:26:02.750133 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:02.750136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:02.750194 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:02.750274 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:26:02.751379 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:02.752013 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:02.753535 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:02.753745 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:02.753835 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:02.756594 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:26:02.756598 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:26:02.756602 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:26:02.756632 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:26:02.758332 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:02.758916 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:02.758921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:02.758923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:02.759018 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:02.759357 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:02.759364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:02.759368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:02.759644 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:26:02.759771 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:02.760223 (+ 452us) server_negotiation.cc:299] Negotiation successful
0810 07:26:02.760364 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 07:26:02.761226 30204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:02.748089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:40969 (local address 127.12.157.194:35745)
0810 07:26:02.748524 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:02.748565 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 07:26:02.749921 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:26:02.750429 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:26:02.750437 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:26:02.750777 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:26:02.751236 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:02.751248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:02.752167 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:02.752170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:02.753410 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:02.753424 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:02.753898 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:02.753905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:02.754141 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:02.754762 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:26:02.754787 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:26:02.756434 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:26:02.758450 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:26:02.758455 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:26:02.758465 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:26:02.758798 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:26:02.759117 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:26:02.759119 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:26:02.759120 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:26:02.759247 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:26:02.759761 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:26:02.759767 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:26:02.759982 (+ 215us) client_negotiation.cc:769] Sending connection context
0810 07:26:02.760134 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 07:26:02.760381 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":114,"threads_started":1}
I20260810 07:26:02.762327 30201 heartbeater.cc:347] Connected to a master server at 127.12.157.254:40969
I20260810 07:26:02.762578 30201 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.12.157.195@KRBTEST.COM; defaulting to no policy
I20260810 07:26:02.763068 30201 heartbeater.cc:511] Master 127.12.157.254:40969 requested a full tablet report, sending...
I20260810 07:26:02.764154 29749 ts_manager.cc:194] Registered new tserver with Master: 6bafaee674a0461c9f5bff5c7f4c6820 (127.12.157.194:33123)
I20260810 07:26:02.764740 29749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.194@KRBTEST.COM'} at 127.12.157.194:35745
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
I20260810 07:26:02.782162 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:0
--local_ip_for_outbound_sockets=127.12.157.195
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:40969
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:34539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:26:02.897590 30208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:26:02.897876 30208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:26:02.897921 30208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:26:02.902020 30208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:26:02.902077 30208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:26:02.902161 30208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:26:02.907586 30208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:34539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:40969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:26:02.908694 30208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:26:02.909808 30208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:26:02.920023 30216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:26:02.920049 30215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:26:02.920044 30220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:26:02.920348 30208 server_base.cc:1034] running on GCE node
I20260810 07:26:02.920820 30208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:26:02.921523 30208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:26:02.922696 30208 hybrid_clock.cc:648] HybridClock initialized: now 1786346762922672 us; error 33 us; skew 500 ppm
Aug 10 07:26:02 dist-test-slave-ffrd krb5kdc[29685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346762, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:26:02.925571 30208 init.cc:377] Logged in from keytab as kudu/127.12.157.195@KRBTEST.COM (short username kudu)
I20260810 07:26:02.926434 30208 webserver.cc:460] Webserver started at http://127.12.157.195:43729/ using document root <none> and password file <none>
I20260810 07:26:02.926992 30208 fs_manager.cc:359] Metadata directory not provided
I20260810 07:26:02.927055 30208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:26:02.927294 30208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:26:02.929446 30208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/data/instance:
uuid: "1b1cb5cf0fc24402bac5026e9081a78b"
format_stamp: "Formatted at 2026-08-10 07:26:02 on dist-test-slave-ffrd"
server_key: "6d40fac5a80e748a994e8307df744c5b"
server_key_iv: "3f4f1ac52bb14c35fbc806ed8dfcd58d"
server_key_version: "encryptionkey@0"
I20260810 07:26:02.930177 30208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance:
uuid: "1b1cb5cf0fc24402bac5026e9081a78b"
format_stamp: "Formatted at 2026-08-10 07:26:02 on dist-test-slave-ffrd"
server_key: "6d40fac5a80e748a994e8307df744c5b"
server_key_iv: "3f4f1ac52bb14c35fbc806ed8dfcd58d"
server_key_version: "encryptionkey@0"
I20260810 07:26:02.934273 30208 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 07:26:02.937001 30234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:02.938243 30208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:26:02.938409 30208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "1b1cb5cf0fc24402bac5026e9081a78b"
format_stamp: "Formatted at 2026-08-10 07:26:02 on dist-test-slave-ffrd"
server_key: "6d40fac5a80e748a994e8307df744c5b"
server_key_iv: "3f4f1ac52bb14c35fbc806ed8dfcd58d"
server_key_version: "encryptionkey@0"
I20260810 07:26:02.938542 30208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:26:02.966503 30208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:26:02.969797 30208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:26:02.970026 30208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:26:02.970659 30208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:26:02.971650 30208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:26:02.971742 30208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:02.971837 30208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:26:02.971884 30208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:02.981920 30208 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:38677
I20260810 07:26:02.982009 30412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:38677 every 8 connection(s)
I20260810 07:26:02.983026 30208 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:26:02.990010 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 30208
I20260810 07:26:02.990118 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance
I20260810 07:26:02.990424 12919 external_mini_cluster.cc:1428] Setting key 476ad0ef82245ea0b364a92df55e6671
Aug 10 07:26:02 dist-test-slave-ffrd krb5kdc[29685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346762, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:26:02.997218 29809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:02.984892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:59287 (local address 127.12.157.254:40969)
0810 07:26:02.985080 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:02.985085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:02.985964 (+ 879us) server_negotiation.cc:407] Connection header received
0810 07:26:02.987333 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:02.987336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:02.987396 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:02.987473 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:26:02.988662 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:02.989290 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:02.990576 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:02.990856 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:02.990968 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:02.993800 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:26:02.993804 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:26:02.993806 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:26:02.993829 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:26:02.995093 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:02.995609 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:02.995614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:02.995616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:02.995698 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:02.996054 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:02.996060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:02.996061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:02.996245 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:26:02.996354 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:02.996886 (+ 532us) server_negotiation.cc:299] Negotiation successful
0810 07:26:02.997049 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 07:26:02.998035 30417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:02.985291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:40969 (local address 127.12.157.195:59287)
0810 07:26:02.985802 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:02.985846 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 07:26:02.987139 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:26:02.987634 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:26:02.987640 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:26:02.988019 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:26:02.988496 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:02.988510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:02.989417 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:02.989419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:02.990420 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:02.990430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:02.991028 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:02.991032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:02.991271 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:02.991968 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:26:02.991985 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:26:02.993632 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:26:02.995220 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:26:02.995225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:26:02.995239 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:26:02.995504 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:26:02.995801 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:26:02.995804 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:26:02.995806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:26:02.995957 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:26:02.996354 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:26:02.996362 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:26:02.996629 (+ 267us) client_negotiation.cc:769] Sending connection context
0810 07:26:02.996832 (+ 203us) client_negotiation.cc:240] Negotiation successful
0810 07:26:02.997185 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":118,"threads_started":1}
I20260810 07:26:02.999121 30414 heartbeater.cc:347] Connected to a master server at 127.12.157.254:40969
I20260810 07:26:02.999413 30414 heartbeater.cc:464] Registering TS with master...
I20260810 07:26:02.999958 30414 heartbeater.cc:511] Master 127.12.157.254:40969 requested a full tablet report, sending...
I20260810 07:26:03.001127 29749 ts_manager.cc:194] Registered new tserver with Master: 1b1cb5cf0fc24402bac5026e9081a78b (127.12.157.195:38677)
I20260810 07:26:03.001729 29749 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.195@KRBTEST.COM'} at 127.12.157.195:59287
I20260810 07:26:03.004585 12919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:26:03.014655 29809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:03.006748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32850 (local address 127.12.157.254:40969)
0810 07:26:03.006923 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:03.006927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:03.007125 (+ 198us) server_negotiation.cc:407] Connection header received
0810 07:26:03.007191 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:03.007194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:03.007238 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:03.007306 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:26:03.008030 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:03.008547 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:03.009461 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:03.009677 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:03.009766 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:03.011126 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:26:03.011129 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:26:03.011131 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:26:03.011157 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:26:03.012793 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:03.013340 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:03.013343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:03.013344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:03.013448 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:03.013784 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:03.013788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:03.013789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:03.014037 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:26:03.014153 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:03.014419 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 07:26:03.014525 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 07:26:03.018296 29749 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:26:03.021819 29749 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:26:03.040129 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:03.034591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:46429 (local address 127.0.0.1:35914)
0810 07:26:03.035103 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:03.035117 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:26:03.035207 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:26:03.036105 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:26:03.036108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:26:03.036125 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:26:03.036218 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:03.036226 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:03.037495 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:03.037501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:03.038493 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:03.038504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:03.039789 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:03.039794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:03.039881 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:03.039903 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 07:26:03.039968 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 07:26:03.040021 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":102,"threads_started":1}
I20260810 07:26:03.040158 30432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:03.034187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47346 (local address 127.12.157.194:33123)
0810 07:26:03.034703 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:03.034717 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:03.034748 (+ 31us) server_negotiation.cc:407] Connection header received
0810 07:26:03.034837 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:03.034842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:03.035064 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:03.035222 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:26:03.035750 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:03.036895 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:03.038584 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:03.039377 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:03.039509 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:03.039703 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:03.039859 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 07:26:03.039968 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":149,"threads_started":1}
I20260810 07:26:03.040141 30431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:03.034075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:33123 (local address 127.0.0.1:47346)
0810 07:26:03.034494 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:03.034548 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 07:26:03.034686 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:26:03.035444 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:26:03.035450 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:26:03.035488 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:26:03.035628 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:03.035635 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:03.037063 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:03.037067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:03.038442 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:03.038452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:03.039517 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:03.039524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:03.039635 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:03.039674 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 07:26:03.039815 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 07:26:03.039954 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":1,"spinlock_wait_cycles":7424,"thread_start_us":114,"threads_started":1}
I20260810 07:26:03.040885 30436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:03.035087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35914 (local address 127.12.157.193:46429)
0810 07:26:03.035507 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:03.035515 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:03.035537 (+ 22us) server_negotiation.cc:407] Connection header received
0810 07:26:03.035612 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:03.035617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:03.035822 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:03.035961 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:26:03.036366 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:03.037378 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:03.038642 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:03.039467 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:03.039585 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:03.039717 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:03.040603 (+ 886us) server_negotiation.cc:299] Negotiation successful
0810 07:26:03.040734 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":115,"threads_started":1}
I20260810 07:26:03.040887 30434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:03.034848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:38677 (local address 127.0.0.1:33534)
0810 07:26:03.035389 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:03.035401 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:26:03.035497 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:26:03.036519 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:26:03.036522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:26:03.036537 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:26:03.036610 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:03.036614 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:03.038033 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:03.038037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:03.039158 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:03.039168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:03.040454 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:03.040457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:03.040556 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:03.040578 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 07:26:03.040640 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 07:26:03.040739 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":1,"spinlock_wait_cycles":2048,"thread_start_us":94,"threads_started":1}
I20260810 07:26:03.041541 30439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:03.035401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33534 (local address 127.12.157.195:38677)
0810 07:26:03.035936 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:03.035947 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:03.035968 (+ 21us) server_negotiation.cc:407] Connection header received
0810 07:26:03.036044 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:03.036048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:03.036214 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:03.036376 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:26:03.036726 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:03.037883 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:03.039295 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:03.040184 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:03.040318 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:03.041151 (+ 833us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:03.041203 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 07:26:03.041294 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":107,"threads_started":1}
I20260810 07:26:03.043967 29926 tablet_service.cc:1467] Processing CreateTablet for tablet 409737d5aae64160a1daa269c11da21c (DEFAULT_TABLE table=test-table [id=d9cdbf4898224cff9a18e243a3012c07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:26:03.043965 30103 tablet_service.cc:1467] Processing CreateTablet for tablet 409737d5aae64160a1daa269c11da21c (DEFAULT_TABLE table=test-table [id=d9cdbf4898224cff9a18e243a3012c07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:26:03.044642 30309 tablet_service.cc:1467] Processing CreateTablet for tablet 409737d5aae64160a1daa269c11da21c (DEFAULT_TABLE table=test-table [id=d9cdbf4898224cff9a18e243a3012c07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:26:03.045218 30103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 409737d5aae64160a1daa269c11da21c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:26:03.045218 29926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 409737d5aae64160a1daa269c11da21c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:26:03.045598 30309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 409737d5aae64160a1daa269c11da21c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:26:03.052815 30440 tablet_bootstrap.cc:492] T 409737d5aae64160a1daa269c11da21c P 1b1cb5cf0fc24402bac5026e9081a78b: Bootstrap starting.
I20260810 07:26:03.053670 30442 tablet_bootstrap.cc:492] T 409737d5aae64160a1daa269c11da21c P 6bafaee674a0461c9f5bff5c7f4c6820: Bootstrap starting.
I20260810 07:26:03.055739 30440 tablet_bootstrap.cc:654] T 409737d5aae64160a1daa269c11da21c P 1b1cb5cf0fc24402bac5026e9081a78b: Neither blocks nor log segments found. Creating new log.
I20260810 07:26:03.056173 30442 tablet_bootstrap.cc:654] T 409737d5aae64160a1daa269c11da21c P 6bafaee674a0461c9f5bff5c7f4c6820: Neither blocks nor log segments found. Creating new log.
I20260810 07:26:03.056929 30440 log.cc:824] T 409737d5aae64160a1daa269c11da21c P 1b1cb5cf0fc24402bac5026e9081a78b: Log is configured to *not* fsync() on all Append() calls
I20260810 07:26:03.057246 30442 log.cc:824] T 409737d5aae64160a1daa269c11da21c P 6bafaee674a0461c9f5bff5c7f4c6820: Log is configured to *not* fsync() on all Append() calls
I20260810 07:26:03.058040 30441 tablet_bootstrap.cc:492] T 409737d5aae64160a1daa269c11da21c P 65b5fd650d1c46d5b985968e1ca2eba9: Bootstrap starting.
I20260810 07:26:03.059249 30442 tablet_bootstrap.cc:492] T 409737d5aae64160a1daa269c11da21c P 6bafaee674a0461c9f5bff5c7f4c6820: No bootstrap required, opened a new log
I20260810 07:26:03.059250 30440 tablet_bootstrap.cc:492] T 409737d5aae64160a1daa269c11da21c P 1b1cb5cf0fc24402bac5026e9081a78b: No bootstrap required, opened a new log
I20260810 07:26:03.059463 30442 ts_tablet_manager.cc:1397] T 409737d5aae64160a1daa269c11da21c P 6bafaee674a0461c9f5bff5c7f4c6820: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 07:26:03.059463 30440 ts_tablet_manager.cc:1397] T 409737d5aae64160a1daa269c11da21c P 1b1cb5cf0fc24402bac5026e9081a78b: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20260810 07:26:03.060091 30441 tablet_bootstrap.cc:654] T 409737d5aae64160a1daa269c11da21c P 65b5fd650d1c46d5b985968e1ca2eba9: Neither blocks nor log segments found. Creating new log.
I20260810 07:26:03.060966 30441 log.cc:824] T 409737d5aae64160a1daa269c11da21c P 65b5fd650d1c46d5b985968e1ca2eba9: Log is configured to *not* fsync() on all Append() calls
I20260810 07:26:03.062686 30441 tablet_bootstrap.cc:492] T 409737d5aae64160a1daa269c11da21c P 65b5fd650d1c46d5b985968e1ca2eba9: No bootstrap required, opened a new log
I20260810 07:26:03.062898 30441 ts_tablet_manager.cc:1397] T 409737d5aae64160a1daa269c11da21c P 65b5fd650d1c46d5b985968e1ca2eba9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 07:26:03.063577 30440 raft_consensus.cc:348] T 409737d5aae64160a1daa269c11da21c P 1b1cb5cf0fc24402bac5026e9081a78b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bafaee674a0461c9f5bff5c7f4c6820" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 33123 } } peers { permanent_uuid: "65b5fd650d1c46d5b985968e1ca2eba9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46429 } } peers { permanent_uuid: "1b1cb5cf0fc24402bac5026e9081a78b" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 38677 } }
I20260810 07:26:03.063577 30442 raft_consensus.cc:348] T 409737d5aae64160a1daa269c11da21c P 6bafaee674a0461c9f5bff5c7f4c6820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bafaee674a0461c9f5bff5c7f4c6820" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 33123 } } peers { permanent_uuid: "65b5fd650d1c46d5b985968e1ca2eba9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46429 } } peers { permanent_uuid: "1b1cb5cf0fc24402bac5026e9081a78b" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 38677 } }
I20260810 07:26:03.063862 30442 raft_consensus.cc:374] T 409737d5aae64160a1daa269c11da21c P 6bafaee674a0461c9f5bff5c7f4c6820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:26:03.063862 30440 raft_consensus.cc:374] T 409737d5aae64160a1daa269c11da21c P 1b1cb5cf0fc24402bac5026e9081a78b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:26:03.063925 30442 raft_consensus.cc:729] T 409737d5aae64160a1daa269c11da21c P 6bafaee674a0461c9f5bff5c7f4c6820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bafaee674a0461c9f5bff5c7f4c6820, State: Initialized, Role: FOLLOWER
I20260810 07:26:03.063936 30440 raft_consensus.cc:729] T 409737d5aae64160a1daa269c11da21c P 1b1cb5cf0fc24402bac5026e9081a78b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b1cb5cf0fc24402bac5026e9081a78b, State: Initialized, Role: FOLLOWER
I20260810 07:26:03.064405 30440 consensus_queue.cc:260] T 409737d5aae64160a1daa269c11da21c P 1b1cb5cf0fc24402bac5026e9081a78b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bafaee674a0461c9f5bff5c7f4c6820" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 33123 } } peers { permanent_uuid: "65b5fd650d1c46d5b985968e1ca2eba9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46429 } } peers { permanent_uuid: "1b1cb5cf0fc24402bac5026e9081a78b" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 38677 } }
I20260810 07:26:03.064405 30442 consensus_queue.cc:260] T 409737d5aae64160a1daa269c11da21c P 6bafaee674a0461c9f5bff5c7f4c6820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bafaee674a0461c9f5bff5c7f4c6820" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 33123 } } peers { permanent_uuid: "65b5fd650d1c46d5b985968e1ca2eba9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46429 } } peers { permanent_uuid: "1b1cb5cf0fc24402bac5026e9081a78b" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 38677 } }
I20260810 07:26:03.065393 30201 heartbeater.cc:503] Master 127.12.157.254:40969 was elected leader, sending a full tablet report...
I20260810 07:26:03.065410 30414 heartbeater.cc:503] Master 127.12.157.254:40969 was elected leader, sending a full tablet report...
I20260810 07:26:03.065481 30440 ts_tablet_manager.cc:1428] T 409737d5aae64160a1daa269c11da21c P 1b1cb5cf0fc24402bac5026e9081a78b: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260810 07:26:03.065481 30442 ts_tablet_manager.cc:1428] T 409737d5aae64160a1daa269c11da21c P 6bafaee674a0461c9f5bff5c7f4c6820: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 07:26:03.070609 30441 raft_consensus.cc:348] T 409737d5aae64160a1daa269c11da21c P 65b5fd650d1c46d5b985968e1ca2eba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bafaee674a0461c9f5bff5c7f4c6820" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 33123 } } peers { permanent_uuid: "65b5fd650d1c46d5b985968e1ca2eba9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46429 } } peers { permanent_uuid: "1b1cb5cf0fc24402bac5026e9081a78b" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 38677 } }
I20260810 07:26:03.070943 30441 raft_consensus.cc:374] T 409737d5aae64160a1daa269c11da21c P 65b5fd650d1c46d5b985968e1ca2eba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:26:03.071027 30441 raft_consensus.cc:729] T 409737d5aae64160a1daa269c11da21c P 65b5fd650d1c46d5b985968e1ca2eba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b5fd650d1c46d5b985968e1ca2eba9, State: Initialized, Role: FOLLOWER
I20260810 07:26:03.071533 30441 consensus_queue.cc:260] T 409737d5aae64160a1daa269c11da21c P 65b5fd650d1c46d5b985968e1ca2eba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bafaee674a0461c9f5bff5c7f4c6820" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 33123 } } peers { permanent_uuid: "65b5fd650d1c46d5b985968e1ca2eba9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46429 } } peers { permanent_uuid: "1b1cb5cf0fc24402bac5026e9081a78b" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 38677 } }
I20260810 07:26:03.072289 30013 heartbeater.cc:503] Master 127.12.157.254:40969 was elected leader, sending a full tablet report...
I20260810 07:26:03.072556 30441 ts_tablet_manager.cc:1428] T 409737d5aae64160a1daa269c11da21c P 65b5fd650d1c46d5b985968e1ca2eba9: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
W20260810 07:26:03.234021 30416 tablet.cc:2369] T 409737d5aae64160a1daa269c11da21c P 1b1cb5cf0fc24402bac5026e9081a78b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:26:03.239176 30452 raft_consensus.cc:482] T 409737d5aae64160a1daa269c11da21c P 65b5fd650d1c46d5b985968e1ca2eba9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:26:03.239425 30452 raft_consensus.cc:504] T 409737d5aae64160a1daa269c11da21c P 65b5fd650d1c46d5b985968e1ca2eba9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bafaee674a0461c9f5bff5c7f4c6820" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 33123 } } peers { permanent_uuid: "65b5fd650d1c46d5b985968e1ca2eba9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46429 } } peers { permanent_uuid: "1b1cb5cf0fc24402bac5026e9081a78b" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 38677 } }
I20260810 07:26:03.240679 30452 leader_election.cc:288] T 409737d5aae64160a1daa269c11da21c P 65b5fd650d1c46d5b985968e1ca2eba9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bafaee674a0461c9f5bff5c7f4c6820 (127.12.157.194:33123), 1b1cb5cf0fc24402bac5026e9081a78b (127.12.157.195:38677)
I20260810 07:26:03.246062 30439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:03.241655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:35807 (local address 127.12.157.195:38677)
0810 07:26:03.241806 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:03.241811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:03.241911 (+ 100us) server_negotiation.cc:407] Connection header received
0810 07:26:03.242022 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:03.242025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:03.242074 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:03.242153 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:26:03.242379 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:03.243136 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:03.244710 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:03.245384 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:03.245514 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:03.245569 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:03.245854 (+ 285us) server_negotiation.cc:299] Negotiation successful
0810 07:26:03.245931 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 07:26:03.246062 30432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:03.241095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:46903 (local address 127.12.157.194:33123)
0810 07:26:03.241274 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:03.241279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:03.241506 (+ 227us) server_negotiation.cc:407] Connection header received
0810 07:26:03.241656 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:03.241660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:03.241722 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:03.241810 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:26:03.242103 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:03.242969 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:03.244710 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:03.245384 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:03.245514 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:03.245569 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:03.245854 (+ 285us) server_negotiation.cc:299] Negotiation successful
0810 07:26:03.245931 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 07:26:03.246140 30455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:03.241530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:38677 (local address 127.12.157.193:35807)
0810 07:26:03.241812 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:03.241825 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:26:03.241935 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:26:03.242162 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:26:03.242164 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:26:03.242180 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:26:03.242265 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:03.242269 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:03.243210 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:03.243213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:03.244548 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:03.244560 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:03.245520 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:03.245524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:03.245679 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:03.245741 (+ 62us) client_negotiation.cc:769] Sending connection context
0810 07:26:03.245821 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 07:26:03.245943 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":110,"thread_start_us":106,"threads_started":1}
I20260810 07:26:03.246140 30453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:03.241006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:33123 (local address 127.12.157.193:46903)
0810 07:26:03.241369 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:03.241383 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:26:03.241530 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:26:03.241819 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:26:03.241823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:26:03.241841 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:26:03.241985 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:03.241991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:03.243077 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:03.243080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:03.244556 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:03.244561 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:03.245520 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:03.245524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:03.245695 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:03.245743 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 07:26:03.245821 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 07:26:03.245939 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":144,"thread_start_us":136,"threads_started":1}
I20260810 07:26:03.246825 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409737d5aae64160a1daa269c11da21c" candidate_uuid: "65b5fd650d1c46d5b985968e1ca2eba9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bafaee674a0461c9f5bff5c7f4c6820" is_pre_election: true
I20260810 07:26:03.246971 30347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409737d5aae64160a1daa269c11da21c" candidate_uuid: "65b5fd650d1c46d5b985968e1ca2eba9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b1cb5cf0fc24402bac5026e9081a78b" is_pre_election: true
W20260810 07:26:03.247083 30203 tablet.cc:2369] T 409737d5aae64160a1daa269c11da21c P 6bafaee674a0461c9f5bff5c7f4c6820: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:26:03.247170 30135 raft_consensus.cc:2454] T 409737d5aae64160a1daa269c11da21c P 6bafaee674a0461c9f5bff5c7f4c6820 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b5fd650d1c46d5b985968e1ca2eba9 in term 0.
I20260810 07:26:03.247275 30347 raft_consensus.cc:2454] T 409737d5aae64160a1daa269c11da21c P 1b1cb5cf0fc24402bac5026e9081a78b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65b5fd650d1c46d5b985968e1ca2eba9 in term 0.
I20260810 07:26:03.247645 29864 leader_election.cc:302] T 409737d5aae64160a1daa269c11da21c P 65b5fd650d1c46d5b985968e1ca2eba9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b5fd650d1c46d5b985968e1ca2eba9, 6bafaee674a0461c9f5bff5c7f4c6820; no voters:
I20260810 07:26:03.247936 30452 raft_consensus.cc:2781] T 409737d5aae64160a1daa269c11da21c P 65b5fd650d1c46d5b985968e1ca2eba9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:26:03.248014 30452 raft_consensus.cc:482] T 409737d5aae64160a1daa269c11da21c P 65b5fd650d1c46d5b985968e1ca2eba9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:26:03.248046 30452 raft_consensus.cc:3037] T 409737d5aae64160a1daa269c11da21c P 65b5fd650d1c46d5b985968e1ca2eba9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:26:03.249205 30452 raft_consensus.cc:504] T 409737d5aae64160a1daa269c11da21c P 65b5fd650d1c46d5b985968e1ca2eba9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bafaee674a0461c9f5bff5c7f4c6820" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 33123 } } peers { permanent_uuid: "65b5fd650d1c46d5b985968e1ca2eba9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46429 } } peers { permanent_uuid: "1b1cb5cf0fc24402bac5026e9081a78b" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 38677 } }
I20260810 07:26:03.249542 30452 leader_election.cc:288] T 409737d5aae64160a1daa269c11da21c P 65b5fd650d1c46d5b985968e1ca2eba9 [CANDIDATE]: Term 1 election: Requested vote from peers 6bafaee674a0461c9f5bff5c7f4c6820 (127.12.157.194:33123), 1b1cb5cf0fc24402bac5026e9081a78b (127.12.157.195:38677)
I20260810 07:26:03.249953 30135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409737d5aae64160a1daa269c11da21c" candidate_uuid: "65b5fd650d1c46d5b985968e1ca2eba9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bafaee674a0461c9f5bff5c7f4c6820"
I20260810 07:26:03.250046 30347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "409737d5aae64160a1daa269c11da21c" candidate_uuid: "65b5fd650d1c46d5b985968e1ca2eba9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b1cb5cf0fc24402bac5026e9081a78b"
I20260810 07:26:03.250089 30135 raft_consensus.cc:3037] T 409737d5aae64160a1daa269c11da21c P 6bafaee674a0461c9f5bff5c7f4c6820 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:26:03.250171 30347 raft_consensus.cc:3037] T 409737d5aae64160a1daa269c11da21c P 1b1cb5cf0fc24402bac5026e9081a78b [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:26:03.251418 30135 raft_consensus.cc:2454] T 409737d5aae64160a1daa269c11da21c P 6bafaee674a0461c9f5bff5c7f4c6820 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b5fd650d1c46d5b985968e1ca2eba9 in term 1.
I20260810 07:26:03.251418 30347 raft_consensus.cc:2454] T 409737d5aae64160a1daa269c11da21c P 1b1cb5cf0fc24402bac5026e9081a78b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65b5fd650d1c46d5b985968e1ca2eba9 in term 1.
I20260810 07:26:03.251693 29864 leader_election.cc:302] T 409737d5aae64160a1daa269c11da21c P 65b5fd650d1c46d5b985968e1ca2eba9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b1cb5cf0fc24402bac5026e9081a78b, 65b5fd650d1c46d5b985968e1ca2eba9; no voters:
I20260810 07:26:03.251849 30452 raft_consensus.cc:2781] T 409737d5aae64160a1daa269c11da21c P 65b5fd650d1c46d5b985968e1ca2eba9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:26:03.252059 30452 raft_consensus.cc:686] T 409737d5aae64160a1daa269c11da21c P 65b5fd650d1c46d5b985968e1ca2eba9 [term 1 LEADER]: Becoming Leader. State: Replica: 65b5fd650d1c46d5b985968e1ca2eba9, State: Running, Role: LEADER
I20260810 07:26:03.252377 30452 consensus_queue.cc:237] T 409737d5aae64160a1daa269c11da21c P 65b5fd650d1c46d5b985968e1ca2eba9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bafaee674a0461c9f5bff5c7f4c6820" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 33123 } } peers { permanent_uuid: "65b5fd650d1c46d5b985968e1ca2eba9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46429 } } peers { permanent_uuid: "1b1cb5cf0fc24402bac5026e9081a78b" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 38677 } }
I20260810 07:26:03.255685 29748 catalog_manager.cc:5495] T 409737d5aae64160a1daa269c11da21c P 65b5fd650d1c46d5b985968e1ca2eba9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65b5fd650d1c46d5b985968e1ca2eba9 (127.12.157.193). New cstate: current_term: 1 leader_uuid: "65b5fd650d1c46d5b985968e1ca2eba9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bafaee674a0461c9f5bff5c7f4c6820" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 33123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65b5fd650d1c46d5b985968e1ca2eba9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b1cb5cf0fc24402bac5026e9081a78b" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 38677 } health_report { overall_health: UNKNOWN } } }
W20260810 07:26:03.266775 30014 tablet.cc:2369] T 409737d5aae64160a1daa269c11da21c P 65b5fd650d1c46d5b985968e1ca2eba9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:26:03.281023 30436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:03.277491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35920 (local address 127.12.157.193:46429)
0810 07:26:03.277631 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:03.277635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:03.277651 (+ 16us) server_negotiation.cc:407] Connection header received
0810 07:26:03.277740 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:03.277742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:03.277798 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:03.277909 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:26:03.278176 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:03.278675 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:03.279541 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:03.279762 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:03.279873 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:03.280193 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:26:03.280591 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:26:03.280728 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:03.280778 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 07:26:03.280838 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 07:26:03.286990 30135 raft_consensus.cc:1261] T 409737d5aae64160a1daa269c11da21c P 6bafaee674a0461c9f5bff5c7f4c6820 [term 1 FOLLOWER]: Refusing update from remote peer 65b5fd650d1c46d5b985968e1ca2eba9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:26:03.286998 30347 raft_consensus.cc:1261] T 409737d5aae64160a1daa269c11da21c P 1b1cb5cf0fc24402bac5026e9081a78b [term 1 FOLLOWER]: Refusing update from remote peer 65b5fd650d1c46d5b985968e1ca2eba9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:26:03.287772 30456 consensus_queue.cc:1035] T 409737d5aae64160a1daa269c11da21c P 65b5fd650d1c46d5b985968e1ca2eba9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bafaee674a0461c9f5bff5c7f4c6820" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:26:03.287983 30452 consensus_queue.cc:1035] T 409737d5aae64160a1daa269c11da21c P 65b5fd650d1c46d5b985968e1ca2eba9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b1cb5cf0fc24402bac5026e9081a78b" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 38677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:26:03.295689 30459 mvcc.cc:205] Tried to move back new op lower bound from 7316876342415675392 to 7316876342284660736. Current Snapshot: MvccSnapshot[applied={T|T < 7316876342415675392}]
I20260810 07:26:03.296178 30461 mvcc.cc:205] Tried to move back new op lower bound from 7316876342415675392 to 7316876342284660736. Current Snapshot: MvccSnapshot[applied={T|T < 7316876342415675392}]
I20260810 07:26:03.297108 30464 mvcc.cc:205] Tried to move back new op lower bound from 7316876342415675392 to 7316876342284660736. Current Snapshot: MvccSnapshot[applied={T|T < 7316876342415675392}]
I20260810 07:26:03.305207 29749 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:26:03.305425 29749 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:26:03.308023 29749 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a45081e31b534a74944723e890924707: Sending DeleteTablet for 3 replicas of tablet 409737d5aae64160a1daa269c11da21c
I20260810 07:26:03.308902 29926 tablet_service.cc:1514] Processing DeleteTablet for tablet 409737d5aae64160a1daa269c11da21c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:26:03 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:35914
I20260810 07:26:03.308976 30309 tablet_service.cc:1514] Processing DeleteTablet for tablet 409737d5aae64160a1daa269c11da21c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:26:03 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:33534
I20260810 07:26:03.309520 30478 tablet_replica.cc:330] T 409737d5aae64160a1daa269c11da21c P 1b1cb5cf0fc24402bac5026e9081a78b: stopping tablet replica
I20260810 07:26:03.309552 30103 tablet_service.cc:1514] Processing DeleteTablet for tablet 409737d5aae64160a1daa269c11da21c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:26:03 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:47346
I20260810 07:26:03.310077 30481 tablet_replica.cc:330] T 409737d5aae64160a1daa269c11da21c P 65b5fd650d1c46d5b985968e1ca2eba9: stopping tablet replica
I20260810 07:26:03.310163 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 29844
I20260810 07:26:03.310047 30478 raft_consensus.cc:2229] T 409737d5aae64160a1daa269c11da21c P 1b1cb5cf0fc24402bac5026e9081a78b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:26:03.325258 30482 tablet_replica.cc:330] T 409737d5aae64160a1daa269c11da21c P 6bafaee674a0461c9f5bff5c7f4c6820: stopping tablet replica
I20260810 07:26:03.325371 30478 raft_consensus.cc:2258] T 409737d5aae64160a1daa269c11da21c P 1b1cb5cf0fc24402bac5026e9081a78b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:26:03.325886 30482 raft_consensus.cc:2229] T 409737d5aae64160a1daa269c11da21c P 6bafaee674a0461c9f5bff5c7f4c6820 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:26:03.326280 30482 raft_consensus.cc:2258] T 409737d5aae64160a1daa269c11da21c P 6bafaee674a0461c9f5bff5c7f4c6820 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:26:03.328104 30478 ts_tablet_manager.cc:1905] T 409737d5aae64160a1daa269c11da21c P 1b1cb5cf0fc24402bac5026e9081a78b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:26:03.330559 30482 ts_tablet_manager.cc:1905] T 409737d5aae64160a1daa269c11da21c P 6bafaee674a0461c9f5bff5c7f4c6820: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:26:03.331499 30478 ts_tablet_manager.cc:1918] T 409737d5aae64160a1daa269c11da21c P 1b1cb5cf0fc24402bac5026e9081a78b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:26:03.331626 30478 log.cc:1196] T 409737d5aae64160a1daa269c11da21c P 1b1cb5cf0fc24402bac5026e9081a78b: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-2/wal/wals/409737d5aae64160a1daa269c11da21c
I20260810 07:26:03.332043 30478 ts_tablet_manager.cc:1939] T 409737d5aae64160a1daa269c11da21c P 1b1cb5cf0fc24402bac5026e9081a78b: Deleting consensus metadata
I20260810 07:26:03.333798 29727 catalog_manager.cc:4841] TS 1b1cb5cf0fc24402bac5026e9081a78b (127.12.157.195:38677): tablet 409737d5aae64160a1daa269c11da21c (table test-table [id=d9cdbf4898224cff9a18e243a3012c07]) successfully deleted
I20260810 07:26:03.333899 30482 ts_tablet_manager.cc:1918] T 409737d5aae64160a1daa269c11da21c P 6bafaee674a0461c9f5bff5c7f4c6820: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:26:03.333992 30482 log.cc:1196] T 409737d5aae64160a1daa269c11da21c P 6bafaee674a0461c9f5bff5c7f4c6820: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786346694975122-12919-0/minicluster-data/ts-1/wal/wals/409737d5aae64160a1daa269c11da21c
I20260810 07:26:03.334335 30482 ts_tablet_manager.cc:1939] T 409737d5aae64160a1daa269c11da21c P 6bafaee674a0461c9f5bff5c7f4c6820: Deleting consensus metadata
I20260810 07:26:03.335461 29727 catalog_manager.cc:4841] TS 6bafaee674a0461c9f5bff5c7f4c6820 (127.12.157.194:33123): tablet 409737d5aae64160a1daa269c11da21c (table test-table [id=d9cdbf4898224cff9a18e243a3012c07]) successfully deleted
W20260810 07:26:03.339241 29726 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.157.193:46429 (error 108)
I20260810 07:26:03.339363 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 30020
I20260810 07:26:03.345031 30434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:03.344315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:46429 (local address 127.0.0.1:35926)
0810 07:26:03.344746 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:03.344864 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.193:46429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312}
W20260810 07:26:03.346577 29726 catalog_manager.cc:4584] TS 65b5fd650d1c46d5b985968e1ca2eba9 (127.12.157.193:46429): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 409737d5aae64160a1daa269c11da21c: Network error: Client connection negotiation failed: client connection to 127.12.157.193:46429: connect: Connection refused (error 111)
I20260810 07:26:03.357575 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 30208
I20260810 07:26:03.371025 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 29702
2026-08-10T07:26:03Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2769 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:26:03 dist-test-slave-ffrd krb5kdc[30487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:26:03 dist-test-slave-ffrd krb5kdc[30487](info): set up 2 sockets
Aug 10 07:26:03 dist-test-slave-ffrd krb5kdc[30487](info): commencing operation
krb5kdc: starting...
W20260810 07:26:04.650626 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.248s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:26:04 dist-test-slave-ffrd krb5kdc[30487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:26:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:26:04Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.157.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.157.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.157.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/rangeradmin_127.12.157.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.157.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/rangeradmin_127.12.157.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.157.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.157.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.157.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/rangerlookup_127.12.157.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.157.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/rangerlookup_127.12.157.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.157.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.157.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.157.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/HTTP_127.12.157.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.157.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/HTTP_127.12.157.212@KRBTEST.COM.keytab.
I20260810 07:26:04.829962 12919 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasko6sLg5/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/postgres -l logfile start
2026-08-10 07:26:07.463 UTC [30529] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 07:26:07.463 UTC [30529] LOG: listening on IPv4 address "127.12.157.212", port 48529
2026-08-10 07:26:07.467 UTC [30529] LOG: listening on Unix socket "/tmp/.s.PGSQL.48529"
2026-08-10 07:26:07.491 UTC [30533] LOG: database system was shut down at 2026-08-10 07:26:05 UTC
2026-08-10 07:26:07.496 UTC [30529] LOG: database system is ready to accept connections
I20260810 07:26:08.753852 12919 mini_postgres.cc:96] Postgres bound to 48529
2026-08-10 07:26:08.763 UTC [30546] FATAL: database "slave" does not exist
127.12.157.212:48529 - accepting connections
I20260810 07:26:08.764297 12919 mini_ranger.cc:161] Starting Ranger...
I20260810 07:26:08.781529 12919 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 07:26:09.291054 12919 mini_ranger.cc:87] Created ranger Postgres database
I20260810 07:26:09.291164 12919 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ranger-admin
2026-08-10 07:26:09,651 [I] DB FLAVOR :POSTGRES
2026-08-10 07:26:09,652 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 07:26:09,652 [I] Checking connection..
2026-08-10 07:26:09,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:48529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 07:26:09,926 [I] Checking connection passed.
2026-08-10 07:26:09,927 [I] --------- Verifying version history table ---------
2026-08-10 07:26:09,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:48529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 07:26:10,191 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 07:26:10,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:48529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 07:26:10,467 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 07:26:10,467 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 07:26:10,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:48529/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 07:26:10.726 UTC [30636] WARNING: there is no transaction in progress
2026-08-10 07:26:10,734 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 07:26:10,734 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:48529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 07:26:10,996 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 07:26:10,996 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 07:26:10,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:48529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 07:26:11,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:48529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 07:26:11,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:48529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-ffrd.c.gcp-upstream.internal','N') ;"
2026-08-10 07:26:11,755 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:48529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 07:26:12,026 [I] Table x_portal_user does not exist in database ranger
2026-08-10 07:26:12,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:48529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 07:26:12,297 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 07:26:12,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:48529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 07:26:12,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:48529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 07:26:12,797 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 07:26:12,798 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:48529/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 07:26:13.855 UTC [30792] WARNING: there is no transaction in progress
2026-08-10 07:26:13.872 UTC [30792] WARNING: there is no transaction in progress
2026-08-10 07:26:13.889 UTC [30792] WARNING: there is no transaction in progress
2026-08-10 07:26:13.905 UTC [30792] WARNING: there is no transaction in progress
2026-08-10 07:26:13.920 UTC [30792] WARNING: there is no transaction in progress
2026-08-10 07:26:13.936 UTC [30792] WARNING: there is no transaction in progress
2026-08-10 07:26:13.989 UTC [30792] WARNING: there is no transaction in progress
2026-08-10 07:26:13.996 UTC [30792] WARNING: there is no transaction in progress
2026-08-10 07:26:14.011 UTC [30792] WARNING: there is no transaction in progress
2026-08-10 07:26:14.022 UTC [30792] WARNING: there is no transaction in progress
2026-08-10 07:26:14.033 UTC [30792] WARNING: there is no transaction in progress
2026-08-10 07:26:14.048 UTC [30792] WARNING: there is no transaction in progress
2026-08-10 07:26:14.059 UTC [30792] WARNING: there is no transaction in progress
2026-08-10 07:26:14.065 UTC [30792] WARNING: there is no transaction in progress
2026-08-10 07:26:14.072 UTC [30792] WARNING: there is no transaction in progress
2026-08-10 07:26:14.079 UTC [30792] WARNING: there is no transaction in progress
2026-08-10 07:26:14,756 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 07:26:14,756 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:48529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 07:26:15,025 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 07:26:15,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:48529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 07:26:15,305 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 07:26:15,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:48529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 07:26:15,564 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 07:26:15,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:48529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 07:26:15,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:48529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 07:26:16,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:48529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-ffrd.c.gcp-upstream.internal';"
2026-08-10 07:26:16,321 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 07:26:16,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:48529/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 07:26:16,567 [I] DB_PATCHES have already been applied
I20260810 07:26:16.573794 12919 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ranger-admin:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasko6sLg5/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 07:26:16.573901 12919 mini_ranger.cc:193] Using host: 127.12.157.212
I20260810 07:26:16.578550 12919 mini_ranger.cc:227] Ranger admin URL: http://127.12.157.212:48975
Aug 10, 2026 7:26:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 7:26:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 7:26:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 7:26:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 7:26:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 7:26:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 7:26:17 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 07:26:17.964253 12919 mini_ranger.cc:160] Time spent starting Ranger: real 9.200s user 0.002s sys 0.011s
Aug 10, 2026 7:26:39 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 7:26:39 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 7:26:39 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 7:26:39 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 7:26:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 7:26:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 7:26:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 7:26:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 7:26:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 7:26:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 7:26:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 7:26:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 7:26:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 7:26:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 7:26:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 7:26:39 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 7:26:39 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 7:26:40 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 07:26:40.591616 12919 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.157.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.157.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.157.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/rangerkms_127.12.157.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.157.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/rangerkms_127.12.157.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.157.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.157.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.157.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/HTTP_127.12.157.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.157.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/HTTP_127.12.157.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 07:26:40 dist-test-slave-ffrd krb5kdc[30487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346800, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 07:26:40.681761 12919 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 07:26:40.691931 12919 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 07:26:41.541837 12919 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 07:26:41.541949 12919 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ranger-kms
I20260810 07:26:41.542073 12919 mini_ranger_kms.cc:227] Using postgres at 127.12.157.212:48529
2026-08-10 07:26:41,860 [I] DB FLAVOR :POSTGRES
2026-08-10 07:26:41,860 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 07:26:41,860 [I] Checking connection
2026-08-10 07:26:41,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:48529/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 07:26:42,102 [I] connection success
2026-08-10 07:26:42,102 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 07:26:42,102 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 07:26:42,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:48529/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 07:26:42,364 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 07:26:42,365 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 07:26:42,365 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 07:26:42,365 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:48529/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 07:26:42,647 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 07:26:43.167426 12919 mini_ranger_kms.cc:310] Created kms service
I20260810 07:26:43.343880 12919 mini_ranger_kms.cc:326] Created kudu user
I20260810 07:26:43.394021 12919 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 07:26:43 dist-test-slave-ffrd krb5kdc[30487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346800, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.157.212@KRBTEST.COM
I20260810 07:26:43.875007 12919 mini_ranger_kms.cc:384] Added ranger policy
I20260810 07:26:43.875190 12919 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ranger-kms:/tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 07:26:43.875218 12919 mini_ranger_kms.cc:240] Using host: 127.12.157.212
I20260810 07:26:43.880511 12919 mini_ranger_kms.cc:276] Ranger KMS PID: 31076
I20260810 07:26:43.880666 12919 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.157.212:58243
Aug 10, 2026 7:26:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 7:26:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 7:26:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 7:26:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 7:26:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 7:26:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 07:26:46 dist-test-slave-ffrd krb5kdc[30487](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786346806, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.157.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:26:46.675194 12919 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.993s user 0.006s sys 0.296s
I20260810 07:26:46.675360 12919 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 07:26:46.675487 12919 mini_ranger_kms.cc:401] 127.12.157.212:58243/kms/v1/keys
Aug 10 07:26:46 dist-test-slave-ffrd krb5kdc[30487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346806, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.157.212@KRBTEST.COM for HTTP/127.12.157.212@KRBTEST.COM
Aug 10 07:26:47 dist-test-slave-ffrd krb5kdc[30487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346806, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.157.212@KRBTEST.COM for HTTP/127.12.157.212@KRBTEST.COM
Aug 10 07:26:49 dist-test-slave-ffrd krb5kdc[30487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:26:49.622289 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:41723
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:41723
--ranger_config_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.157.212:58243/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:26:49.761019 31154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:26:49.761307 31154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:26:49.761348 31154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:26:49.765580 31154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:26:49.765645 31154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:26:49.765667 31154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:26:49.765684 31154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:26:49.765717 31154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:26:49.771328 31154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.157.212:58243/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:41723
--ranger_config_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:41723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:26:49.772408 31154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:26:49.773519 31154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:26:49.783464 31161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:26:49.783535 31154 server_base.cc:1034] running on GCE node
W20260810 07:26:49.783475 31162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:26:49.783895 31164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:26:49.784556 31154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:26:49.786051 31154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:26:49.787256 31154 hybrid_clock.cc:648] HybridClock initialized: now 1786346809787226 us; error 42 us; skew 500 ppm
Aug 10 07:26:49 dist-test-slave-ffrd krb5kdc[30487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346809, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:26:49.790895 31154 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:26:49.791917 31154 webserver.cc:460] Webserver started at http://127.12.157.254:34965/ using document root <none> and password file <none>
I20260810 07:26:49.792490 31154 fs_manager.cc:359] Metadata directory not provided
I20260810 07:26:49.792572 31154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:26:49.792815 31154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 07:26:49 dist-test-slave-ffrd krb5kdc[30487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346809, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for HTTP/127.12.157.212@KRBTEST.COM
I20260810 07:26:49.843719 31154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "0f85cb6971884f87b481dc18f2f4df4f"
format_stamp: "Formatted at 2026-08-10 07:26:49 on dist-test-slave-ffrd"
server_key: "5c28e928817e89f6552b04905d0aecb9"
server_key_iv: "c0a4e4eeb2d08f458b87513b36eee8f0"
server_key_version: "kuduclusterkey@0"
I20260810 07:26:49.844342 31154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "0f85cb6971884f87b481dc18f2f4df4f"
format_stamp: "Formatted at 2026-08-10 07:26:49 on dist-test-slave-ffrd"
server_key: "5c28e928817e89f6552b04905d0aecb9"
server_key_iv: "c0a4e4eeb2d08f458b87513b36eee8f0"
server_key_version: "kuduclusterkey@0"
I20260810 07:26:49.848356 31154 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 07:26:49.867447 31174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:49.868871 31154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 07:26:49.869052 31154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "0f85cb6971884f87b481dc18f2f4df4f"
format_stamp: "Formatted at 2026-08-10 07:26:49 on dist-test-slave-ffrd"
server_key: "5c28e928817e89f6552b04905d0aecb9"
server_key_iv: "c0a4e4eeb2d08f458b87513b36eee8f0"
server_key_version: "kuduclusterkey@0"
I20260810 07:26:49.869208 31154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:26:49.899048 31154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:26:49.903379 31154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:26:49.903843 31154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:26:49.911995 31154 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:41723
I20260810 07:26:49.912011 31246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:41723 every 8 connection(s)
I20260810 07:26:49.913208 31154 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:26:49.916177 31247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:26:49.922885 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 31154
I20260810 07:26:49.923007 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:26:49.923300 12919 external_mini_cluster.cc:1428] Setting key 7602c302ab54a3dc7f012eba7720c693
I20260810 07:26:49.923743 31247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f85cb6971884f87b481dc18f2f4df4f: Bootstrap starting.
I20260810 07:26:49.926594 31247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f85cb6971884f87b481dc18f2f4df4f: Neither blocks nor log segments found. Creating new log.
I20260810 07:26:49.927349 31247 log.cc:824] T 00000000000000000000000000000000 P 0f85cb6971884f87b481dc18f2f4df4f: Log is configured to *not* fsync() on all Append() calls
I20260810 07:26:49.929322 31247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f85cb6971884f87b481dc18f2f4df4f: No bootstrap required, opened a new log
Aug 10 07:26:49 dist-test-slave-ffrd krb5kdc[30487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:26:49.934105 31247 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f85cb6971884f87b481dc18f2f4df4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f85cb6971884f87b481dc18f2f4df4f" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41723 } }
I20260810 07:26:49.934377 31247 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f85cb6971884f87b481dc18f2f4df4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:26:49.934445 31247 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f85cb6971884f87b481dc18f2f4df4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f85cb6971884f87b481dc18f2f4df4f, State: Initialized, Role: FOLLOWER
I20260810 07:26:49.934933 31247 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f85cb6971884f87b481dc18f2f4df4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f85cb6971884f87b481dc18f2f4df4f" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41723 } }
I20260810 07:26:49.935060 31247 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f85cb6971884f87b481dc18f2f4df4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:26:49.935153 31247 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f85cb6971884f87b481dc18f2f4df4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:26:49.935263 31247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f85cb6971884f87b481dc18f2f4df4f [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:26:49.936498 31247 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f85cb6971884f87b481dc18f2f4df4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f85cb6971884f87b481dc18f2f4df4f" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41723 } }
I20260810 07:26:49.936873 31247 leader_election.cc:302] T 00000000000000000000000000000000 P 0f85cb6971884f87b481dc18f2f4df4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f85cb6971884f87b481dc18f2f4df4f; no voters:
I20260810 07:26:49.937316 31247 leader_election.cc:288] T 00000000000000000000000000000000 P 0f85cb6971884f87b481dc18f2f4df4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:26:49.937443 31253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f85cb6971884f87b481dc18f2f4df4f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:26:49.937695 31253 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f85cb6971884f87b481dc18f2f4df4f [term 1 LEADER]: Becoming Leader. State: Replica: 0f85cb6971884f87b481dc18f2f4df4f, State: Running, Role: LEADER
I20260810 07:26:49.938112 31253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f85cb6971884f87b481dc18f2f4df4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f85cb6971884f87b481dc18f2f4df4f" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41723 } }
I20260810 07:26:49.938688 31247 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f85cb6971884f87b481dc18f2f4df4f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:26:49.939158 31251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:49.924866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51482 (local address 127.12.157.254:41723)
0810 07:26:49.925424 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:49.925442 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:49.925481 (+ 39us) server_negotiation.cc:407] Connection header received
0810 07:26:49.926184 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:49.926208 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:49.926593 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:49.926958 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:26:49.927807 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:49.928801 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:49.929760 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:49.930242 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:49.930445 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:49.933490 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:26:49.933496 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:26:49.933512 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:26:49.933544 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:26:49.935548 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:49.936064 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:49.936068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:49.936074 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:49.936183 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:49.936515 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:49.936518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:49.936519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:49.937114 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:26:49.937283 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:49.937687 (+ 404us) server_negotiation.cc:299] Negotiation successful
0810 07:26:49.938189 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":153,"threads_started":1}
I20260810 07:26:49.939461 31253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f85cb6971884f87b481dc18f2f4df4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f85cb6971884f87b481dc18f2f4df4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f85cb6971884f87b481dc18f2f4df4f" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41723 } } }
I20260810 07:26:49.939612 31253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f85cb6971884f87b481dc18f2f4df4f [sys.catalog]: This master's current role is: LEADER
I20260810 07:26:49.939628 31254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f85cb6971884f87b481dc18f2f4df4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f85cb6971884f87b481dc18f2f4df4f. Latest consensus state: current_term: 1 leader_uuid: "0f85cb6971884f87b481dc18f2f4df4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f85cb6971884f87b481dc18f2f4df4f" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41723 } } }
I20260810 07:26:49.939726 31254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f85cb6971884f87b481dc18f2f4df4f [sys.catalog]: This master's current role is: LEADER
I20260810 07:26:49.941301 31247 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 07:26:49.942092 31268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:26:49.948414 31268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:26:49.956265 31268 catalog_manager.cc:1347] Generated new cluster ID: c52df5e9243b48bebc6f3d65077a1f49
I20260810 07:26:49.956470 31268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:26:49.982530 31268 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:26:49.984164 31268 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:26:49.999519 31268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f85cb6971884f87b481dc18f2f4df4f: Generated new TSK 0
I20260810 07:26:50.000608 31268 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
I20260810 07:26:50.719503 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.157.212:58243/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:0
--local_ip_for_outbound_sockets=127.12.157.193
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:41723
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:26:50.847508 31307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:26:50.847813 31307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:26:50.847895 31307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:26:50.852326 31307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:26:50.852427 31307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:26:50.852511 31307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:26:50.858778 31307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.157.212:58243/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:41723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:26:50.860024 31307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:26:50.861644 31307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:26:50.873857 31321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:26:50.873809 31322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:26:50.874249 31307 server_base.cc:1034] running on GCE node
W20260810 07:26:50.874537 31324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:26:50.875012 31307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:26:50.875737 31307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:26:50.876977 31307 hybrid_clock.cc:648] HybridClock initialized: now 1786346810876911 us; error 49 us; skew 500 ppm
Aug 10 07:26:50 dist-test-slave-ffrd krb5kdc[30487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346810, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:26:50.880340 31307 init.cc:377] Logged in from keytab as kudu/127.12.157.193@KRBTEST.COM (short username kudu)
I20260810 07:26:50.881462 31307 webserver.cc:460] Webserver started at http://127.12.157.193:37067/ using document root <none> and password file <none>
I20260810 07:26:50.882119 31307 fs_manager.cc:359] Metadata directory not provided
I20260810 07:26:50.882244 31307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:26:50.882543 31307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 07:26:50 dist-test-slave-ffrd krb5kdc[30487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346810, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for HTTP/127.12.157.212@KRBTEST.COM
I20260810 07:26:50.898913 31307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-0/data/instance:
uuid: "209b0234087541d590b746581df1f73b"
format_stamp: "Formatted at 2026-08-10 07:26:50 on dist-test-slave-ffrd"
server_key: "d18b2f89b09c3922dfab71b072d1b093"
server_key_iv: "9f5ed028bd71270f4a82c36cbea0c2c6"
server_key_version: "kuduclusterkey@0"
I20260810 07:26:50.899691 31307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance:
uuid: "209b0234087541d590b746581df1f73b"
format_stamp: "Formatted at 2026-08-10 07:26:50 on dist-test-slave-ffrd"
server_key: "d18b2f89b09c3922dfab71b072d1b093"
server_key_iv: "9f5ed028bd71270f4a82c36cbea0c2c6"
server_key_version: "kuduclusterkey@0"
I20260810 07:26:50.905196 31307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260810 07:26:50.930326 31334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:50.931756 31307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 07:26:50.931928 31307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "209b0234087541d590b746581df1f73b"
format_stamp: "Formatted at 2026-08-10 07:26:50 on dist-test-slave-ffrd"
server_key: "d18b2f89b09c3922dfab71b072d1b093"
server_key_iv: "9f5ed028bd71270f4a82c36cbea0c2c6"
server_key_version: "kuduclusterkey@0"
I20260810 07:26:50.932065 31307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:26:50.952380 31307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:26:50.956113 31307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:26:50.956317 31307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:26:50.957070 31307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:26:50.958200 31307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:26:50.958309 31307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:50.958443 31307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:26:50.958534 31307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:50.969566 31307 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:39737
I20260810 07:26:50.970772 31307 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:26:50.973598 31493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:39737 every 8 connection(s)
I20260810 07:26:50.980517 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 31307
I20260810 07:26:50.980657 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance
I20260810 07:26:50.981006 12919 external_mini_cluster.cc:1428] Setting key fba105a39ab61308f5815b9a58fb9ab9
Aug 10 07:26:50 dist-test-slave-ffrd krb5kdc[30487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346810, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.157.194@KRBTEST.COM; defaulting to no policy
I20260810 07:26:51.004791 31497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:50.976382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:41723 (local address 127.12.157.193:53947)
0810 07:26:50.978687 (+ 2305us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:50.978774 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0810 07:26:50.980427 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:26:50.982493 (+ 2066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:26:50.982504 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:26:50.982901 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:26:50.983681 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:50.983707 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:50.991739 (+ 8032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:50.991744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:50.993394 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:50.993416 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:50.994400 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:50.994406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:50.994656 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:50.995487 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:26:50.995512 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:26:50.997430 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:26:51.001408 (+ 3978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:26:51.001416 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:26:51.001438 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:26:51.001844 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:26:51.002221 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:26:51.002224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:26:51.002226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:26:51.002402 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:26:51.002960 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:26:51.002970 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:26:51.003325 (+ 355us) client_negotiation.cc:769] Sending connection context
0810 07:26:51.003563 (+ 238us) client_negotiation.cc:240] Negotiation successful
0810 07:26:51.003887 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1943,"thread_start_us":150,"threads_started":1}
I20260810 07:26:51.005702 31499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:50.977531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:53947 (local address 127.12.157.254:41723)
0810 07:26:50.981900 (+ 4369us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:50.981910 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:50.981934 (+ 24us) server_negotiation.cc:407] Connection header received
0810 07:26:50.981973 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:50.981976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:50.982020 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:50.982113 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:26:50.983959 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:50.991484 (+ 7525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:50.993612 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:50.994030 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:50.994150 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:50.997698 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:26:50.997702 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:26:50.997705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:26:50.997738 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:26:51.001260 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:51.002006 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:51.002009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:51.002011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:51.002101 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:51.002525 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:51.002528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:51.002530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:51.002805 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:26:51.005355 (+ 2550us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:51.005456 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 07:26:51.005580 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4286,"thread_start_us":120,"threads_started":1}
I20260810 07:26:51.006945 31495 heartbeater.cc:347] Connected to a master server at 127.12.157.254:41723
I20260810 07:26:51.007243 31495 heartbeater.cc:464] Registering TS with master...
I20260810 07:26:51.007934 31495 heartbeater.cc:511] Master 127.12.157.254:41723 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.194@KRBTEST.COM" created.
I20260810 07:26:51.009960 31196 ts_manager.cc:194] Registered new tserver with Master: 209b0234087541d590b746581df1f73b (127.12.157.193:39737)
I20260810 07:26:51.011595 31196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.193@KRBTEST.COM'} at 127.12.157.193:53947
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
I20260810 07:26:51.030474 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.157.212:58243/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:0
--local_ip_for_outbound_sockets=127.12.157.194
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:41723
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 07:26:51 dist-test-slave-ffrd krb5kdc[30487](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786346811, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 07:26:51.186916 31505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:26:51.187357 31505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:26:51.187433 31505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:26:51.191917 31505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:26:51.192034 31505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:26:51.192147 31505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:26:51.199875 31505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.157.212:58243/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:41723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:26:51.201184 31505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:26:51.202276 31505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:26:51.213135 31515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:26:51.213135 31514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:26:51.213562 31519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:26:51.214154 31505 server_base.cc:1034] running on GCE node
I20260810 07:26:51.214620 31505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:26:51.215611 31505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:26:51.216830 31505 hybrid_clock.cc:648] HybridClock initialized: now 1786346811216807 us; error 79 us; skew 500 ppm
Aug 10 07:26:51 dist-test-slave-ffrd krb5kdc[30487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346811, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:26:51.221060 31505 init.cc:377] Logged in from keytab as kudu/127.12.157.194@KRBTEST.COM (short username kudu)
I20260810 07:26:51.222159 31505 webserver.cc:460] Webserver started at http://127.12.157.194:40979/ using document root <none> and password file <none>
I20260810 07:26:51.222747 31505 fs_manager.cc:359] Metadata directory not provided
I20260810 07:26:51.222851 31505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:26:51.223093 31505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 07:26:51 dist-test-slave-ffrd krb5kdc[30487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346811, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for HTTP/127.12.157.212@KRBTEST.COM
I20260810 07:26:51.240561 31505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-1/data/instance:
uuid: "e338dc8d02eb427da3e265885b87194d"
format_stamp: "Formatted at 2026-08-10 07:26:51 on dist-test-slave-ffrd"
server_key: "fe0c2e3949f79d966ab81ed58f24dbd7"
server_key_iv: "dac29d09eb689ccb81574cb94663b6c2"
server_key_version: "kuduclusterkey@0"
I20260810 07:26:51.241518 31505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance:
uuid: "e338dc8d02eb427da3e265885b87194d"
format_stamp: "Formatted at 2026-08-10 07:26:51 on dist-test-slave-ffrd"
server_key: "fe0c2e3949f79d966ab81ed58f24dbd7"
server_key_iv: "dac29d09eb689ccb81574cb94663b6c2"
server_key_version: "kuduclusterkey@0"
I20260810 07:26:51.247179 31505 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 07:26:51.269443 31528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:51.271270 31505 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.000s
I20260810 07:26:51.271428 31505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "e338dc8d02eb427da3e265885b87194d"
format_stamp: "Formatted at 2026-08-10 07:26:51 on dist-test-slave-ffrd"
server_key: "fe0c2e3949f79d966ab81ed58f24dbd7"
server_key_iv: "dac29d09eb689ccb81574cb94663b6c2"
server_key_version: "kuduclusterkey@0"
I20260810 07:26:51.271580 31505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:26:51.302044 31505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:26:51.305403 31505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:26:51.305621 31505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:26:51.306336 31505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:26:51.307292 31505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:26:51.307361 31505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:51.307471 31505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:26:51.307514 31505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:51.319502 31505 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:40465
I20260810 07:26:51.319715 31683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:40465 every 8 connection(s)
I20260810 07:26:51.320703 31505 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:26:51.326391 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 31505
I20260810 07:26:51.326525 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance
I20260810 07:26:51.326828 12919 external_mini_cluster.cc:1428] Setting key d426041363ddb7bc409234ffa50ef1fd
Aug 10 07:26:51 dist-test-slave-ffrd krb5kdc[30487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346811, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:26:51.338456 31688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:51.323321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:41723 (local address 127.12.157.194:52245)
0810 07:26:51.323812 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:51.323867 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 07:26:51.324747 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:26:51.325325 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:26:51.325332 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:26:51.325602 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:26:51.326117 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:51.326140 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:51.327623 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:51.327626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:51.329014 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:51.329029 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:51.330297 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:51.330303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:51.330540 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:51.331317 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:26:51.331344 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:26:51.333168 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:26:51.335025 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:26:51.335031 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:26:51.335047 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:26:51.335407 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:26:51.335858 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:26:51.335861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:26:51.335863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:26:51.336043 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:26:51.336559 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:26:51.336566 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:26:51.336907 (+ 341us) client_negotiation.cc:769] Sending connection context
0810 07:26:51.337162 (+ 255us) client_negotiation.cc:240] Negotiation successful
0810 07:26:51.337533 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":146,"threads_started":1}
I20260810 07:26:51.339255 31499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:51.322898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:52245 (local address 127.12.157.254:41723)
0810 07:26:51.323064 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:51.323069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:51.323972 (+ 903us) server_negotiation.cc:407] Connection header received
0810 07:26:51.324983 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:51.324987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:51.325045 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:51.325143 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:26:51.326322 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:51.327097 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:51.329159 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:51.329542 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:51.329639 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:51.333338 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:26:51.333342 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:26:51.333345 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:26:51.333370 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:26:51.334888 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:51.335533 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:51.335537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:51.335538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:51.335628 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:51.336160 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:51.336164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:51.336166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:51.336397 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:26:51.336496 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:51.338895 (+ 2399us) server_negotiation.cc:299] Negotiation successful
0810 07:26:51.339033 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 07:26:51.341343 31685 heartbeater.cc:347] Connected to a master server at 127.12.157.254:41723
I20260810 07:26:51.341598 31685 heartbeater.cc:464] Registering TS with master...
I20260810 07:26:51.342222 31685 heartbeater.cc:511] Master 127.12.157.254:41723 requested a full tablet report, sending...
I20260810 07:26:51.343488 31196 ts_manager.cc:194] Registered new tserver with Master: e338dc8d02eb427da3e265885b87194d (127.12.157.194:40465)
I20260810 07:26:51.344153 31196 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.194@KRBTEST.COM'} at 127.12.157.194:52245
WARNING: no policy specified for kudu/127.12.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
I20260810 07:26:51.370321 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.157.212:58243/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:0
--local_ip_for_outbound_sockets=127.12.157.195
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:41723
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:26:51.533200 31693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:26:51.533594 31693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:26:51.533663 31693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:26:51.540019 31693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:26:51.540119 31693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:26:51.540212 31693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:26:51.548424 31693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.157.212:58243/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:41723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:26:51.549937 31693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:26:51.551363 31693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:26:51.564790 31706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:26:51.565204 31700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:26:51.566515 31702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:26:51.567173 31693 server_base.cc:1034] running on GCE node
I20260810 07:26:51.567862 31693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:26:51.568785 31693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:26:51.569962 31693 hybrid_clock.cc:648] HybridClock initialized: now 1786346811569879 us; error 97 us; skew 500 ppm
Aug 10 07:26:51 dist-test-slave-ffrd krb5kdc[30487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346811, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:26:51.574240 31693 init.cc:377] Logged in from keytab as kudu/127.12.157.195@KRBTEST.COM (short username kudu)
I20260810 07:26:51.575482 31693 webserver.cc:460] Webserver started at http://127.12.157.195:44903/ using document root <none> and password file <none>
I20260810 07:26:51.576278 31693 fs_manager.cc:359] Metadata directory not provided
I20260810 07:26:51.576380 31693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:26:51.576643 31693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 07:26:51 dist-test-slave-ffrd krb5kdc[30487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346811, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for HTTP/127.12.157.212@KRBTEST.COM
Aug 10 07:26:51 dist-test-slave-ffrd krb5kdc[30487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346811, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for HTTP/127.12.157.212@KRBTEST.COM
I20260810 07:26:51.592706 31693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-2/data/instance:
uuid: "3d01dfd20ba748c2acc86c60abba3019"
format_stamp: "Formatted at 2026-08-10 07:26:51 on dist-test-slave-ffrd"
server_key: "f2a5a4b8234ef825d38a5ed19aecb21b"
server_key_iv: "5f95e87dbbe84a08a1330d8c5edf8113"
server_key_version: "kuduclusterkey@0"
I20260810 07:26:51.593552 31693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance:
uuid: "3d01dfd20ba748c2acc86c60abba3019"
format_stamp: "Formatted at 2026-08-10 07:26:51 on dist-test-slave-ffrd"
server_key: "f2a5a4b8234ef825d38a5ed19aecb21b"
server_key_iv: "5f95e87dbbe84a08a1330d8c5edf8113"
server_key_version: "kuduclusterkey@0"
I20260810 07:26:51.605823 31693 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.005s sys 0.000s
I20260810 07:26:51.624390 31717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:51.626232 31693 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260810 07:26:51.626395 31693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "3d01dfd20ba748c2acc86c60abba3019"
format_stamp: "Formatted at 2026-08-10 07:26:51 on dist-test-slave-ffrd"
server_key: "f2a5a4b8234ef825d38a5ed19aecb21b"
server_key_iv: "5f95e87dbbe84a08a1330d8c5edf8113"
server_key_version: "kuduclusterkey@0"
I20260810 07:26:51.626545 31693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:26:51.664502 31693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:26:51.668262 31693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:26:51.668475 31693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:26:51.669245 31693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:26:51.670280 31693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:26:51.670338 31693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:26:51.670389 31693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:26:51.670647 31693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260810 07:26:51.682302 31693 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:45977
I20260810 07:26:51.682464 31869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:45977 every 8 connection(s)
I20260810 07:26:51.683522 31693 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:26:51.690696 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 31693
I20260810 07:26:51.690837 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance
I20260810 07:26:51.691116 12919 external_mini_cluster.cc:1428] Setting key d88f8e920964d20ff9a074fbb0c69831
Aug 10 07:26:51 dist-test-slave-ffrd krb5kdc[30487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346811, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:26:51.700976 31499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:51.685666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:56835 (local address 127.12.157.254:41723)
0810 07:26:51.685800 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:51.685805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:51.686974 (+ 1169us) server_negotiation.cc:407] Connection header received
0810 07:26:51.688220 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:51.688225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:51.688284 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:51.688389 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:26:51.689820 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:51.690693 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:51.692771 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:51.693199 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:51.693277 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:51.696602 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:26:51.696606 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:26:51.696609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:26:51.696638 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:26:51.698417 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:51.699093 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:51.699097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:51.699098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:51.699197 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:51.699617 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:51.699620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:51.699622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:51.699860 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:26:51.699970 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:51.700635 (+ 665us) server_negotiation.cc:299] Negotiation successful
0810 07:26:51.700782 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 07:26:51.702209 31874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:51.686168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:41723 (local address 127.12.157.195:56835)
0810 07:26:51.686783 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:51.686872 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0810 07:26:51.687952 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:26:51.688586 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:26:51.688596 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:26:51.688995 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:26:51.689653 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:51.689671 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:51.690859 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:51.690863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:51.692608 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:51.692620 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:51.693492 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:51.693495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:51.693690 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:51.694473 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:26:51.694496 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:26:51.696391 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:26:51.698563 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:26:51.698570 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:26:51.698586 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:26:51.698958 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:26:51.699299 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:26:51.699303 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:26:51.699305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:26:51.699505 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:26:51.699987 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:26:51.699994 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:26:51.700337 (+ 343us) client_negotiation.cc:769] Sending connection context
0810 07:26:51.700569 (+ 232us) client_negotiation.cc:240] Negotiation successful
0810 07:26:51.700919 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":152,"threads_started":1}
I20260810 07:26:51.703624 31870 heartbeater.cc:347] Connected to a master server at 127.12.157.254:41723
I20260810 07:26:51.703908 31870 heartbeater.cc:464] Registering TS with master...
I20260810 07:26:51.704432 31870 heartbeater.cc:511] Master 127.12.157.254:41723 requested a full tablet report, sending...
I20260810 07:26:51.705530 31195 ts_manager.cc:194] Registered new tserver with Master: 3d01dfd20ba748c2acc86c60abba3019 (127.12.157.195:45977)
I20260810 07:26:51.706293 31195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.195@KRBTEST.COM'} at 127.12.157.195:56835
I20260810 07:26:51.706341 12919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:26:51.717002 31499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:51.709505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51486 (local address 127.12.157.254:41723)
0810 07:26:51.709636 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:51.709640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:51.709721 (+ 81us) server_negotiation.cc:407] Connection header received
0810 07:26:51.709880 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:51.709883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:51.709933 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:51.710013 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:26:51.710717 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:51.711230 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:51.712073 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:51.712293 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:51.712399 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:51.713836 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:26:51.713840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:26:51.713843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:26:51.713868 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:26:51.715319 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:51.715743 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:51.715747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:51.715749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:51.715831 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:26:51.716108 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:26:51.716112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:26:51.716114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:26:51.716289 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:26:51.716428 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:51.716658 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 07:26:51.716773 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 07:26:51.720310 31195 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:26:51.723665 31195 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:26:51.741086 31896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:51.733523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:39737 (local address 127.0.0.1:60928)
0810 07:26:51.735190 (+ 1667us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:51.735241 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 07:26:51.735452 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:26:51.736319 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:26:51.736323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:26:51.736342 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:26:51.736440 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:51.736446 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:51.737814 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:51.737817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:51.739176 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:51.739187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:51.740481 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:51.740484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:51.740591 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:51.740629 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 07:26:51.740739 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 07:26:51.740909 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1500,"mutex_wait_us":1,"thread_start_us":137,"threads_started":1}
I20260810 07:26:51.741811 31897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:51.735294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56224 (local address 127.12.157.194:40465)
0810 07:26:51.736500 (+ 1206us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:51.736514 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:51.736535 (+ 21us) server_negotiation.cc:407] Connection header received
0810 07:26:51.736614 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:51.736620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:51.736789 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:51.737280 (+ 491us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:26:51.737410 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:51.738450 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:51.739871 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:51.740667 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:51.741383 (+ 716us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:51.741507 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:51.741561 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 07:26:51.741654 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1095,"thread_start_us":142,"threads_started":1}
I20260810 07:26:51.741869 31893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:51.733643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60928 (local address 127.12.157.193:39737)
0810 07:26:51.734640 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:51.734648 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:51.735331 (+ 683us) server_negotiation.cc:407] Connection header received
0810 07:26:51.735650 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:51.735655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:51.735887 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:51.736127 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:26:51.736573 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:51.737668 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:51.739317 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:51.740141 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:51.740240 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:51.740389 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:51.741580 (+ 1191us) server_negotiation.cc:299] Negotiation successful
0810 07:26:51.741678 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":881,"thread_start_us":117,"threads_started":1}
I20260810 07:26:51.742027 31892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:51.733593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:45977 (local address 127.0.0.1:45364)
0810 07:26:51.735412 (+ 1819us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:51.735436 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 07:26:51.735617 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:26:51.736077 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:26:51.736082 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:26:51.736102 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:26:51.736230 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:51.736235 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:51.738462 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:51.738466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:51.739652 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:51.739662 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:51.741604 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:51.741608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:51.741692 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:51.741713 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 07:26:51.741798 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 07:26:51.741876 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1722,"mutex_wait_us":3,"thread_start_us":113,"threads_started":1}
I20260810 07:26:51.741269 31891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:51.734781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:40465 (local address 127.0.0.1:56224)
0810 07:26:51.735854 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:51.735866 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:26:51.735942 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:26:51.737028 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:26:51.737031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:26:51.737050 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:26:51.737134 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:51.737140 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:51.738590 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:51.738594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:51.739698 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:51.739709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:51.740816 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:51.740820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:51.740912 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:51.740965 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 07:26:51.741031 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 07:26:51.741103 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1010,"mutex_wait_us":24,"thread_start_us":123,"threads_started":1}
I20260810 07:26:51.742710 31894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:51.734140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45364 (local address 127.12.157.195:45977)
0810 07:26:51.734851 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:51.734866 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:51.735533 (+ 667us) server_negotiation.cc:407] Connection header received
0810 07:26:51.735767 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:51.735772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:51.735918 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:51.736077 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:26:51.736346 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:51.738332 (+ 1986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:51.739854 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:51.741464 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:51.742270 (+ 806us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:51.742405 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:51.742453 (+ 48us) server_negotiation.cc:299] Negotiation successful
0810 07:26:51.742542 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":178,"threads_started":1}
I20260810 07:26:51.745178 31398 tablet_service.cc:1467] Processing CreateTablet for tablet 8f84a1d16396441086d94e9dfb4dc3e8 (DEFAULT_TABLE table=test-table [id=4d6dbcccf6d2473aaf038e362711384a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:26:51.745313 31593 tablet_service.cc:1467] Processing CreateTablet for tablet 8f84a1d16396441086d94e9dfb4dc3e8 (DEFAULT_TABLE table=test-table [id=4d6dbcccf6d2473aaf038e362711384a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:26:51.746241 31398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f84a1d16396441086d94e9dfb4dc3e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:26:51.746507 31593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f84a1d16396441086d94e9dfb4dc3e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:26:51.751335 31756 tablet_service.cc:1467] Processing CreateTablet for tablet 8f84a1d16396441086d94e9dfb4dc3e8 (DEFAULT_TABLE table=test-table [id=4d6dbcccf6d2473aaf038e362711384a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:26:51.752435 31756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f84a1d16396441086d94e9dfb4dc3e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:26:51.755642 31898 tablet_bootstrap.cc:492] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d: Bootstrap starting.
I20260810 07:26:51.758598 31898 tablet_bootstrap.cc:654] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d: Neither blocks nor log segments found. Creating new log.
I20260810 07:26:51.759577 31898 log.cc:824] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d: Log is configured to *not* fsync() on all Append() calls
I20260810 07:26:51.761765 31898 tablet_bootstrap.cc:492] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d: No bootstrap required, opened a new log
I20260810 07:26:51.761827 31900 tablet_bootstrap.cc:492] T 8f84a1d16396441086d94e9dfb4dc3e8 P 3d01dfd20ba748c2acc86c60abba3019: Bootstrap starting.
I20260810 07:26:51.761952 31898 ts_tablet_manager.cc:1397] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20260810 07:26:51.764559 31900 tablet_bootstrap.cc:654] T 8f84a1d16396441086d94e9dfb4dc3e8 P 3d01dfd20ba748c2acc86c60abba3019: Neither blocks nor log segments found. Creating new log.
I20260810 07:26:51.765522 31900 log.cc:824] T 8f84a1d16396441086d94e9dfb4dc3e8 P 3d01dfd20ba748c2acc86c60abba3019: Log is configured to *not* fsync() on all Append() calls
I20260810 07:26:51.765666 31898 raft_consensus.cc:348] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209b0234087541d590b746581df1f73b" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39737 } } peers { permanent_uuid: "3d01dfd20ba748c2acc86c60abba3019" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 45977 } } peers { permanent_uuid: "e338dc8d02eb427da3e265885b87194d" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 40465 } }
I20260810 07:26:51.765933 31898 raft_consensus.cc:374] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:26:51.765977 31898 raft_consensus.cc:729] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e338dc8d02eb427da3e265885b87194d, State: Initialized, Role: FOLLOWER
I20260810 07:26:51.766615 31902 tablet_bootstrap.cc:492] T 8f84a1d16396441086d94e9dfb4dc3e8 P 209b0234087541d590b746581df1f73b: Bootstrap starting.
I20260810 07:26:51.766654 31898 consensus_queue.cc:260] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209b0234087541d590b746581df1f73b" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39737 } } peers { permanent_uuid: "3d01dfd20ba748c2acc86c60abba3019" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 45977 } } peers { permanent_uuid: "e338dc8d02eb427da3e265885b87194d" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 40465 } }
I20260810 07:26:51.767725 31898 ts_tablet_manager.cc:1428] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 07:26:51.768126 31685 heartbeater.cc:503] Master 127.12.157.254:41723 was elected leader, sending a full tablet report...
I20260810 07:26:51.769979 31902 tablet_bootstrap.cc:654] T 8f84a1d16396441086d94e9dfb4dc3e8 P 209b0234087541d590b746581df1f73b: Neither blocks nor log segments found. Creating new log.
I20260810 07:26:51.770802 31902 log.cc:824] T 8f84a1d16396441086d94e9dfb4dc3e8 P 209b0234087541d590b746581df1f73b: Log is configured to *not* fsync() on all Append() calls
I20260810 07:26:51.772653 31900 tablet_bootstrap.cc:492] T 8f84a1d16396441086d94e9dfb4dc3e8 P 3d01dfd20ba748c2acc86c60abba3019: No bootstrap required, opened a new log
I20260810 07:26:51.772866 31900 ts_tablet_manager.cc:1397] T 8f84a1d16396441086d94e9dfb4dc3e8 P 3d01dfd20ba748c2acc86c60abba3019: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260810 07:26:51.774957 31902 tablet_bootstrap.cc:492] T 8f84a1d16396441086d94e9dfb4dc3e8 P 209b0234087541d590b746581df1f73b: No bootstrap required, opened a new log
I20260810 07:26:51.775233 31902 ts_tablet_manager.cc:1397] T 8f84a1d16396441086d94e9dfb4dc3e8 P 209b0234087541d590b746581df1f73b: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260810 07:26:51.778119 31900 raft_consensus.cc:348] T 8f84a1d16396441086d94e9dfb4dc3e8 P 3d01dfd20ba748c2acc86c60abba3019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209b0234087541d590b746581df1f73b" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39737 } } peers { permanent_uuid: "3d01dfd20ba748c2acc86c60abba3019" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 45977 } } peers { permanent_uuid: "e338dc8d02eb427da3e265885b87194d" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 40465 } }
I20260810 07:26:51.778509 31900 raft_consensus.cc:374] T 8f84a1d16396441086d94e9dfb4dc3e8 P 3d01dfd20ba748c2acc86c60abba3019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:26:51.778635 31900 raft_consensus.cc:729] T 8f84a1d16396441086d94e9dfb4dc3e8 P 3d01dfd20ba748c2acc86c60abba3019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d01dfd20ba748c2acc86c60abba3019, State: Initialized, Role: FOLLOWER
I20260810 07:26:51.779225 31900 consensus_queue.cc:260] T 8f84a1d16396441086d94e9dfb4dc3e8 P 3d01dfd20ba748c2acc86c60abba3019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209b0234087541d590b746581df1f73b" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39737 } } peers { permanent_uuid: "3d01dfd20ba748c2acc86c60abba3019" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 45977 } } peers { permanent_uuid: "e338dc8d02eb427da3e265885b87194d" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 40465 } }
I20260810 07:26:51.779644 31902 raft_consensus.cc:348] T 8f84a1d16396441086d94e9dfb4dc3e8 P 209b0234087541d590b746581df1f73b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209b0234087541d590b746581df1f73b" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39737 } } peers { permanent_uuid: "3d01dfd20ba748c2acc86c60abba3019" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 45977 } } peers { permanent_uuid: "e338dc8d02eb427da3e265885b87194d" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 40465 } }
I20260810 07:26:51.779963 31902 raft_consensus.cc:374] T 8f84a1d16396441086d94e9dfb4dc3e8 P 209b0234087541d590b746581df1f73b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:26:51.780054 31902 raft_consensus.cc:729] T 8f84a1d16396441086d94e9dfb4dc3e8 P 209b0234087541d590b746581df1f73b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 209b0234087541d590b746581df1f73b, State: Initialized, Role: FOLLOWER
I20260810 07:26:51.780373 31900 ts_tablet_manager.cc:1428] T 8f84a1d16396441086d94e9dfb4dc3e8 P 3d01dfd20ba748c2acc86c60abba3019: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 07:26:51.780567 31902 consensus_queue.cc:260] T 8f84a1d16396441086d94e9dfb4dc3e8 P 209b0234087541d590b746581df1f73b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209b0234087541d590b746581df1f73b" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39737 } } peers { permanent_uuid: "3d01dfd20ba748c2acc86c60abba3019" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 45977 } } peers { permanent_uuid: "e338dc8d02eb427da3e265885b87194d" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 40465 } }
I20260810 07:26:51.780781 31870 heartbeater.cc:503] Master 127.12.157.254:41723 was elected leader, sending a full tablet report...
I20260810 07:26:51.781585 31902 ts_tablet_manager.cc:1428] T 8f84a1d16396441086d94e9dfb4dc3e8 P 209b0234087541d590b746581df1f73b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 07:26:51.781821 31495 heartbeater.cc:503] Master 127.12.157.254:41723 was elected leader, sending a full tablet report...
W20260810 07:26:51.823167 31687 tablet.cc:2369] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 07:26:51 dist-test-slave-ffrd krb5kdc[30487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346811, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for HTTP/127.12.157.212@KRBTEST.COM
W20260810 07:26:51.935442 31873 tablet.cc:2369] T 8f84a1d16396441086d94e9dfb4dc3e8 P 3d01dfd20ba748c2acc86c60abba3019: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 07:26:51.991843 31496 tablet.cc:2369] T 8f84a1d16396441086d94e9dfb4dc3e8 P 209b0234087541d590b746581df1f73b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:26:52.053458 31905 raft_consensus.cc:482] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:26:52.053722 31905 raft_consensus.cc:504] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209b0234087541d590b746581df1f73b" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39737 } } peers { permanent_uuid: "3d01dfd20ba748c2acc86c60abba3019" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 45977 } } peers { permanent_uuid: "e338dc8d02eb427da3e265885b87194d" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 40465 } }
I20260810 07:26:52.055148 31905 leader_election.cc:288] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 209b0234087541d590b746581df1f73b (127.12.157.193:39737), 3d01dfd20ba748c2acc86c60abba3019 (127.12.157.195:45977)
I20260810 07:26:52.067986 31894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:52.056437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:57911 (local address 127.12.157.195:45977)
0810 07:26:52.056579 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:52.056585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:52.062425 (+ 5840us) server_negotiation.cc:407] Connection header received
0810 07:26:52.062624 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:52.062627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:52.062679 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:52.062730 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:26:52.063172 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:52.063995 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:52.066557 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:52.067208 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:52.067314 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:52.067373 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:52.067727 (+ 354us) server_negotiation.cc:299] Negotiation successful
0810 07:26:52.067834 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 07:26:52.068746 31914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:52.056293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:45977 (local address 127.12.157.194:57911)
0810 07:26:52.062330 (+ 6037us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:52.062344 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:26:52.062499 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:26:52.062892 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:26:52.062895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:26:52.062913 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:26:52.063036 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:52.063043 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:52.064180 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:52.064183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:52.065525 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:52.065537 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:52.067479 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:52.067482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:52.067582 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:52.067605 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 07:26:52.068477 (+ 872us) client_negotiation.cc:240] Negotiation successful
0810 07:26:52.068571 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5970,"thread_start_us":115,"threads_started":1}
I20260810 07:26:52.069679 31794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f84a1d16396441086d94e9dfb4dc3e8" candidate_uuid: "e338dc8d02eb427da3e265885b87194d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d01dfd20ba748c2acc86c60abba3019" is_pre_election: true
I20260810 07:26:52.070098 31794 raft_consensus.cc:2454] T 8f84a1d16396441086d94e9dfb4dc3e8 P 3d01dfd20ba748c2acc86c60abba3019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e338dc8d02eb427da3e265885b87194d in term 0.
I20260810 07:26:52.070725 31530 leader_election.cc:302] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d01dfd20ba748c2acc86c60abba3019, e338dc8d02eb427da3e265885b87194d; no voters:
I20260810 07:26:52.071115 31905 raft_consensus.cc:2781] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:26:52.071203 31905 raft_consensus.cc:482] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:26:52.071244 31905 raft_consensus.cc:3037] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:26:52.072196 31893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:52.056717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:55481 (local address 127.12.157.193:39737)
0810 07:26:52.056876 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:52.056882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:52.062004 (+ 5122us) server_negotiation.cc:407] Connection header received
0810 07:26:52.062160 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:52.062164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:52.062223 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:52.062321 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:26:52.068205 (+ 5884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:52.068989 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:52.070712 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:52.071329 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:52.071425 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:52.071475 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:52.071982 (+ 507us) server_negotiation.cc:299] Negotiation successful
0810 07:26:52.072073 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 07:26:52.072723 31905 raft_consensus.cc:504] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209b0234087541d590b746581df1f73b" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39737 } } peers { permanent_uuid: "3d01dfd20ba748c2acc86c60abba3019" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 45977 } } peers { permanent_uuid: "e338dc8d02eb427da3e265885b87194d" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 40465 } }
I20260810 07:26:52.073012 31913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:52.055677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:39737 (local address 127.12.157.194:55481)
0810 07:26:52.061852 (+ 6175us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:52.061875 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 07:26:52.062047 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:26:52.067737 (+ 5690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:26:52.067741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:26:52.067781 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:26:52.067925 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:52.067931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:52.069229 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:52.069232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:52.070555 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:26:52.070568 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:26:52.071725 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:26:52.071728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:26:52.071840 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:52.071863 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 07:26:52.072677 (+ 814us) client_negotiation.cc:240] Negotiation successful
0810 07:26:52.072785 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6052,"thread_start_us":126,"threads_started":1}
I20260810 07:26:52.073252 31905 leader_election.cc:288] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d [CANDIDATE]: Term 1 election: Requested vote from peers 209b0234087541d590b746581df1f73b (127.12.157.193:39737), 3d01dfd20ba748c2acc86c60abba3019 (127.12.157.195:45977)
I20260810 07:26:52.074286 31794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f84a1d16396441086d94e9dfb4dc3e8" candidate_uuid: "e338dc8d02eb427da3e265885b87194d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d01dfd20ba748c2acc86c60abba3019"
I20260810 07:26:52.074460 31794 raft_consensus.cc:3037] T 8f84a1d16396441086d94e9dfb4dc3e8 P 3d01dfd20ba748c2acc86c60abba3019 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:26:52.074635 31433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f84a1d16396441086d94e9dfb4dc3e8" candidate_uuid: "e338dc8d02eb427da3e265885b87194d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "209b0234087541d590b746581df1f73b" is_pre_election: true
I20260810 07:26:52.074947 31433 raft_consensus.cc:2454] T 8f84a1d16396441086d94e9dfb4dc3e8 P 209b0234087541d590b746581df1f73b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e338dc8d02eb427da3e265885b87194d in term 0.
I20260810 07:26:52.075289 31432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f84a1d16396441086d94e9dfb4dc3e8" candidate_uuid: "e338dc8d02eb427da3e265885b87194d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "209b0234087541d590b746581df1f73b"
I20260810 07:26:52.075387 31432 raft_consensus.cc:3037] T 8f84a1d16396441086d94e9dfb4dc3e8 P 209b0234087541d590b746581df1f73b [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:26:52.075942 31794 raft_consensus.cc:2454] T 8f84a1d16396441086d94e9dfb4dc3e8 P 3d01dfd20ba748c2acc86c60abba3019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e338dc8d02eb427da3e265885b87194d in term 1.
I20260810 07:26:52.076397 31530 leader_election.cc:302] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d01dfd20ba748c2acc86c60abba3019, e338dc8d02eb427da3e265885b87194d; no voters:
I20260810 07:26:52.076604 31432 raft_consensus.cc:2454] T 8f84a1d16396441086d94e9dfb4dc3e8 P 209b0234087541d590b746581df1f73b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e338dc8d02eb427da3e265885b87194d in term 1.
I20260810 07:26:52.076610 31905 raft_consensus.cc:2781] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:26:52.076853 31905 raft_consensus.cc:686] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d [term 1 LEADER]: Becoming Leader. State: Replica: e338dc8d02eb427da3e265885b87194d, State: Running, Role: LEADER
I20260810 07:26:52.077281 31905 consensus_queue.cc:237] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209b0234087541d590b746581df1f73b" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39737 } } peers { permanent_uuid: "3d01dfd20ba748c2acc86c60abba3019" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 45977 } } peers { permanent_uuid: "e338dc8d02eb427da3e265885b87194d" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 40465 } }
I20260810 07:26:52.081288 31195 catalog_manager.cc:5495] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d reported cstate change: term changed from 0 to 1, leader changed from <none> to e338dc8d02eb427da3e265885b87194d (127.12.157.194). New cstate: current_term: 1 leader_uuid: "e338dc8d02eb427da3e265885b87194d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "209b0234087541d590b746581df1f73b" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d01dfd20ba748c2acc86c60abba3019" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 45977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e338dc8d02eb427da3e265885b87194d" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 40465 } health_report { overall_health: HEALTHY } } }
I20260810 07:26:52.119454 31897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:52.113613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56226 (local address 127.12.157.194:40465)
0810 07:26:52.113773 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 07:26:52.113779 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:26:52.113797 (+ 18us) server_negotiation.cc:407] Connection header received
0810 07:26:52.113891 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:26:52.113893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:26:52.113961 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:26:52.114092 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:26:52.114392 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:52.115195 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:52.116704 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:26:52.117300 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:26:52.117657 (+ 357us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:26:52.118050 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:26:52.118649 (+ 599us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:26:52.118767 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 07:26:52.119220 (+ 453us) server_negotiation.cc:299] Negotiation successful
0810 07:26:52.119291 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 07:26:52.126935 31794 raft_consensus.cc:1261] T 8f84a1d16396441086d94e9dfb4dc3e8 P 3d01dfd20ba748c2acc86c60abba3019 [term 1 FOLLOWER]: Refusing update from remote peer e338dc8d02eb427da3e265885b87194d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:26:52.126950 31432 raft_consensus.cc:1261] T 8f84a1d16396441086d94e9dfb4dc3e8 P 209b0234087541d590b746581df1f73b [term 1 FOLLOWER]: Refusing update from remote peer e338dc8d02eb427da3e265885b87194d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:26:52.127709 31905 consensus_queue.cc:1035] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d01dfd20ba748c2acc86c60abba3019" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 45977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:26:52.128095 31915 consensus_queue.cc:1035] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d [LEADER]: Connected to new peer: Peer: permanent_uuid: "209b0234087541d590b746581df1f73b" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:26:52.140158 31919 mvcc.cc:205] Tried to move back new op lower bound from 7316876542463832064 to 7316876542272061440. Current Snapshot: MvccSnapshot[applied={T|T < 7316876542463832064}]
I20260810 07:26:52.140261 31920 mvcc.cc:205] Tried to move back new op lower bound from 7316876542463832064 to 7316876542272061440. Current Snapshot: MvccSnapshot[applied={T|T < 7316876542463832064}]
I20260810 07:26:52.150125 31195 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:26:52.150316 31195 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:26:52.150717 31925 mvcc.cc:205] Tried to move back new op lower bound from 7316876542463832064 to 7316876542272061440. Current Snapshot: MvccSnapshot[applied={T|T < 7316876542463832064}]
I20260810 07:26:52.152902 31195 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f85cb6971884f87b481dc18f2f4df4f: Sending DeleteTablet for 3 replicas of tablet 8f84a1d16396441086d94e9dfb4dc3e8
I20260810 07:26:52.154469 31756 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f84a1d16396441086d94e9dfb4dc3e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:26:52 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:45364
I20260810 07:26:52.154598 31398 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f84a1d16396441086d94e9dfb4dc3e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:26:52 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:60928
I20260810 07:26:52.154992 31593 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f84a1d16396441086d94e9dfb4dc3e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:26:52 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:56224
I20260810 07:26:52.155575 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 31307
I20260810 07:26:52.160724 31933 tablet_replica.cc:330] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d: stopping tablet replica
I20260810 07:26:52.161266 31933 raft_consensus.cc:2229] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:26:52.161983 31933 raft_consensus.cc:2258] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:26:52.166555 31933 ts_tablet_manager.cc:1905] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:26:52.171069 31933 ts_tablet_manager.cc:1918] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:26:52.171222 31933 log.cc:1196] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-1/wal/wals/8f84a1d16396441086d94e9dfb4dc3e8
I20260810 07:26:52.171716 31933 ts_tablet_manager.cc:1939] T 8f84a1d16396441086d94e9dfb4dc3e8 P e338dc8d02eb427da3e265885b87194d: Deleting consensus metadata
I20260810 07:26:52.173534 31179 catalog_manager.cc:4841] TS e338dc8d02eb427da3e265885b87194d (127.12.157.194:40465): tablet 8f84a1d16396441086d94e9dfb4dc3e8 (table test-table [id=4d6dbcccf6d2473aaf038e362711384a]) successfully deleted
I20260810 07:26:52.177207 31931 tablet_replica.cc:330] T 8f84a1d16396441086d94e9dfb4dc3e8 P 3d01dfd20ba748c2acc86c60abba3019: stopping tablet replica
I20260810 07:26:52.184134 31931 raft_consensus.cc:2229] T 8f84a1d16396441086d94e9dfb4dc3e8 P 3d01dfd20ba748c2acc86c60abba3019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:26:52.184731 31931 raft_consensus.cc:2258] T 8f84a1d16396441086d94e9dfb4dc3e8 P 3d01dfd20ba748c2acc86c60abba3019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:26:52.186332 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 31505
W20260810 07:26:52.186357 31175 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.157.193:39737 (error 108)
I20260810 07:26:52.186748 31931 ts_tablet_manager.cc:1905] T 8f84a1d16396441086d94e9dfb4dc3e8 P 3d01dfd20ba748c2acc86c60abba3019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:26:52.200280 31931 ts_tablet_manager.cc:1918] T 8f84a1d16396441086d94e9dfb4dc3e8 P 3d01dfd20ba748c2acc86c60abba3019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:26:52.200430 31931 log.cc:1196] T 8f84a1d16396441086d94e9dfb4dc3e8 P 3d01dfd20ba748c2acc86c60abba3019: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786346694975122-12919-0/minicluster-data/ts-2/wal/wals/8f84a1d16396441086d94e9dfb4dc3e8
I20260810 07:26:52.200785 31931 ts_tablet_manager.cc:1939] T 8f84a1d16396441086d94e9dfb4dc3e8 P 3d01dfd20ba748c2acc86c60abba3019: Deleting consensus metadata
I20260810 07:26:52.201236 31896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:26:52.200663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:39737 (local address 127.0.0.1:60930)
0810 07:26:52.200991 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0810 07:26:52.201101 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.193:39737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
W20260810 07:26:52.201476 31175 catalog_manager.cc:4584] TS 209b0234087541d590b746581df1f73b (127.12.157.193:39737): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8f84a1d16396441086d94e9dfb4dc3e8: Network error: Client connection negotiation failed: client connection to 127.12.157.193:39737: connect: Connection refused (error 111)
I20260810 07:26:52.202275 31176 catalog_manager.cc:4841] TS 3d01dfd20ba748c2acc86c60abba3019 (127.12.157.195:45977): tablet 8f84a1d16396441086d94e9dfb4dc3e8 (table test-table [id=4d6dbcccf6d2473aaf038e362711384a]) successfully deleted
I20260810 07:26:52.231478 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 31693
I20260810 07:26:52.249255 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 31154
I20260810 07:26:52.267637 12919 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 07:26:52.613080 12919 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 07:26:52.613235 12919 mini_ranger.cc:66] Stopping Ranger...
I20260810 07:26:52.959082 12919 mini_ranger.cc:68] Stopped Ranger
2026-08-10 07:26:52.959 UTC [30529] LOG: received smart shutdown request
2026-08-10 07:26:52.963 UTC [30529] LOG: background worker "logical replication launcher" (PID 30543) exited with exit code 1
2026-08-10 07:26:52.963 UTC [30534] LOG: shutting down
2026-08-10 07:26:53.120 UTC [30529] LOG: database system is shut down
2026-08-10T07:26:53Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49865 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:26:53 dist-test-slave-ffrd krb5kdc[31961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:26:53 dist-test-slave-ffrd krb5kdc[31961](info): set up 2 sockets
Aug 10 07:26:53 dist-test-slave-ffrd krb5kdc[31961](info): commencing operation
krb5kdc: starting...
W20260810 07:26:54.526386 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.257s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:26:54 dist-test-slave-ffrd krb5kdc[31961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:26:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:26:54Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.157.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.157.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.157.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/rangeradmin_127.12.157.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.157.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/rangeradmin_127.12.157.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.157.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.157.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.157.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/rangerlookup_127.12.157.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.157.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/rangerlookup_127.12.157.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.157.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.157.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.157.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/HTTP_127.12.157.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.157.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/HTTP_127.12.157.212@KRBTEST.COM.keytab.
I20260810 07:26:54.712101 12919 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasko6sLg5/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/postgres -l logfile start
2026-08-10 07:26:57.159 UTC [32008] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 07:26:57.159 UTC [32008] LOG: listening on IPv4 address "127.12.157.212", port 51987
2026-08-10 07:26:57.162 UTC [32008] LOG: listening on Unix socket "/tmp/.s.PGSQL.51987"
2026-08-10 07:26:57.192 UTC [32011] LOG: database system was shut down at 2026-08-10 07:26:55 UTC
2026-08-10 07:26:57.196 UTC [32008] LOG: database system is ready to accept connections
I20260810 07:26:58.382109 12919 mini_postgres.cc:96] Postgres bound to 51987
2026-08-10 07:26:58.389 UTC [32021] FATAL: database "slave" does not exist
127.12.157.212:51987 - accepting connections
I20260810 07:26:58.390383 12919 mini_ranger.cc:161] Starting Ranger...
I20260810 07:26:58.406584 12919 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 07:26:58.924819 12919 mini_ranger.cc:87] Created ranger Postgres database
I20260810 07:26:58.925091 12919 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ranger-admin
2026-08-10 07:26:59,351 [I] DB FLAVOR :POSTGRES
2026-08-10 07:26:59,351 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 07:26:59,351 [I] Checking connection..
2026-08-10 07:26:59,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:51987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 07:26:59,624 [I] Checking connection passed.
2026-08-10 07:26:59,624 [I] --------- Verifying version history table ---------
2026-08-10 07:26:59,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:51987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 07:26:59,903 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 07:26:59,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:51987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 07:27:00,183 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 07:27:00,183 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 07:27:00,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:51987/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 07:27:00.431 UTC [32113] WARNING: there is no transaction in progress
2026-08-10 07:27:00,438 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 07:27:00,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:51987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 07:27:00,684 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 07:27:00,685 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 07:27:00,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:51987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 07:27:00,949 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:51987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 07:27:01,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:51987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-ffrd.c.gcp-upstream.internal','N') ;"
2026-08-10 07:27:01,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:51987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 07:27:01,726 [I] Table x_portal_user does not exist in database ranger
2026-08-10 07:27:01,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:51987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 07:27:01,969 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 07:27:01,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:51987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 07:27:02,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:51987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 07:27:02,506 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 07:27:02,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:51987/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 07:27:03.527 UTC [32270] WARNING: there is no transaction in progress
2026-08-10 07:27:03.545 UTC [32270] WARNING: there is no transaction in progress
2026-08-10 07:27:03.561 UTC [32270] WARNING: there is no transaction in progress
2026-08-10 07:27:03.577 UTC [32270] WARNING: there is no transaction in progress
2026-08-10 07:27:03.593 UTC [32270] WARNING: there is no transaction in progress
2026-08-10 07:27:03.611 UTC [32270] WARNING: there is no transaction in progress
2026-08-10 07:27:03.669 UTC [32270] WARNING: there is no transaction in progress
2026-08-10 07:27:03.676 UTC [32270] WARNING: there is no transaction in progress
2026-08-10 07:27:03.691 UTC [32270] WARNING: there is no transaction in progress
2026-08-10 07:27:03.701 UTC [32270] WARNING: there is no transaction in progress
2026-08-10 07:27:03.712 UTC [32270] WARNING: there is no transaction in progress
2026-08-10 07:27:03.727 UTC [32270] WARNING: there is no transaction in progress
2026-08-10 07:27:03.737 UTC [32270] WARNING: there is no transaction in progress
2026-08-10 07:27:03.744 UTC [32270] WARNING: there is no transaction in progress
2026-08-10 07:27:03.752 UTC [32270] WARNING: there is no transaction in progress
2026-08-10 07:27:03.759 UTC [32270] WARNING: there is no transaction in progress
2026-08-10 07:27:04,441 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 07:27:04,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:51987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 07:27:04,702 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 07:27:04,702 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:51987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 07:27:04,956 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 07:27:04,957 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:51987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 07:27:05,244 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 07:27:05,245 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:51987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 07:27:05,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:51987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 07:27:05,798 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:51987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-ffrd.c.gcp-upstream.internal';"
2026-08-10 07:27:06,038 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 07:27:06,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:51987/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 07:27:06,301 [I] DB_PATCHES have already been applied
I20260810 07:27:06.307943 12919 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ranger-admin:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasko6sLg5/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 07:27:06.308161 12919 mini_ranger.cc:193] Using host: 127.12.157.212
I20260810 07:27:06.313045 12919 mini_ranger.cc:227] Ranger admin URL: http://127.12.157.212:52109
Aug 10, 2026 7:27:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 7:27:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 7:27:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 7:27:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 7:27:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 7:27:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 7:27:07 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 07:27:07.725144 12919 mini_ranger.cc:160] Time spent starting Ranger: real 9.335s user 0.003s sys 0.015s
Aug 10, 2026 7:27:27 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 7:27:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 7:27:27 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 7:27:27 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 7:27:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 7:27:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 7:27:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 7:27:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 7:27:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 7:27:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 7:27:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 7:27:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 7:27:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 7:27:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 7:27:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 7:27:27 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 7:27:27 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 7:27:28 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 07:27:29.055416 12919 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.157.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.157.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.157.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/rangerkms_127.12.157.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.157.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/rangerkms_127.12.157.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.157.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.157.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.157.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/HTTP_127.12.157.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.157.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/HTTP_127.12.157.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 07:27:29 dist-test-slave-ffrd krb5kdc[31961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346849, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 07:27:29.170322 12919 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 07:27:29.183808 12919 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 07:27:30.066711 12919 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 07:27:30.066834 12919 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ranger-kms
I20260810 07:27:30.066860 12919 mini_ranger_kms.cc:227] Using postgres at 127.12.157.212:51987
2026-08-10 07:27:30,253 [I] DB FLAVOR :POSTGRES
2026-08-10 07:27:30,253 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 07:27:30,253 [I] Checking connection
2026-08-10 07:27:30,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:51987/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 07:27:30,523 [I] connection success
2026-08-10 07:27:30,524 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 07:27:30,524 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 07:27:30,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:51987/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 07:27:30,775 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 07:27:30,775 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 07:27:30,776 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 07:27:30,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.157.212:51987/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 07:27:31,040 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 07:27:31.540439 12919 mini_ranger_kms.cc:310] Created kms service
I20260810 07:27:31.754990 12919 mini_ranger_kms.cc:326] Created kudu user
I20260810 07:27:31.800621 12919 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 07:27:31 dist-test-slave-ffrd krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346849, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.157.212@KRBTEST.COM
I20260810 07:27:32.191332 12919 mini_ranger_kms.cc:384] Added ranger policy
I20260810 07:27:32.191496 12919 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ranger-kms:/tmp/dist-test-tasko6sLg5/build/debug/bin/postgresql.jar:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasko6sLg5/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 07:27:32.191519 12919 mini_ranger_kms.cc:240] Using host: 127.12.157.212
I20260810 07:27:32.196636 12919 mini_ranger_kms.cc:276] Ranger KMS PID: 32525
I20260810 07:27:32.196735 12919 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.157.212:33953
Aug 10, 2026 7:27:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 7:27:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 7:27:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 7:27:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 7:27:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 7:27:33 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 07:27:34 dist-test-slave-ffrd krb5kdc[31961](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786346854, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.157.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 07:27:35 dist-test-slave-ffrd krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346854, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.157.212@KRBTEST.COM for HTTP/127.12.157.212@KRBTEST.COM
I20260810 07:27:35.059818 12919 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.889s user 0.002s sys 0.169s
I20260810 07:27:35.059938 12919 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 07:27:35.060003 12919 mini_ranger_kms.cc:401] 127.12.157.212:33953/kms/v1/keys
Aug 10 07:27:35 dist-test-slave-ffrd krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346854, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.157.212@KRBTEST.COM for HTTP/127.12.157.212@KRBTEST.COM
Aug 10 07:27:37 dist-test-slave-ffrd krb5kdc[31961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:27:37.754391 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:44937
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:44937
--ranger_config_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.157.212:33953/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:37.880411 32603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:37.880689 32603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:37.880730 32603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:37.884824 32603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:27:37.884893 32603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:37.884913 32603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:37.884929 32603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:27:37.885018 32603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:27:37.890626 32603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.157.212:33953/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:44937
--ranger_config_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:44937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:27:37.891718 32603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:37.892783 32603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:37.901283 32609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:37.901278 32612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:37.901278 32610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:37.901664 32603 server_base.cc:1034] running on GCE node
I20260810 07:27:37.902278 32603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:37.903384 32603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:37.904552 32603 hybrid_clock.cc:648] HybridClock initialized: now 1786346857904524 us; error 40 us; skew 500 ppm
Aug 10 07:27:37 dist-test-slave-ffrd krb5kdc[31961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346857, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:37.907474 32603 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:27:37.908351 32603 webserver.cc:460] Webserver started at http://127.12.157.254:40617/ using document root <none> and password file <none>
I20260810 07:27:37.908905 32603 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:37.909015 32603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:37.909240 32603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 07:27:37 dist-test-slave-ffrd krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346857, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for HTTP/127.12.157.212@KRBTEST.COM
I20260810 07:27:37.965098 32603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "36bc2b56025342ec869b7536820e2c80"
format_stamp: "Formatted at 2026-08-10 07:27:37 on dist-test-slave-ffrd"
server_key: "b5ab6e7a2b2c97d1e47a5b09a0080211"
server_key_iv: "d659d38608ceced643678a98d6ff0455"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:37.965770 32603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "36bc2b56025342ec869b7536820e2c80"
format_stamp: "Formatted at 2026-08-10 07:27:37 on dist-test-slave-ffrd"
server_key: "b5ab6e7a2b2c97d1e47a5b09a0080211"
server_key_iv: "d659d38608ceced643678a98d6ff0455"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:37.969905 32603 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 07:27:37.985508 32620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:37.986991 32603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 07:27:37.987172 32603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "36bc2b56025342ec869b7536820e2c80"
format_stamp: "Formatted at 2026-08-10 07:27:37 on dist-test-slave-ffrd"
server_key: "b5ab6e7a2b2c97d1e47a5b09a0080211"
server_key_iv: "d659d38608ceced643678a98d6ff0455"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:37.987325 32603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:38.008765 32603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:38.012123 32603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:38.012375 32603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:38.020967 32603 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:44937
I20260810 07:27:38.021003 32697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:44937 every 8 connection(s)
I20260810 07:27:38.022006 32603 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:27:38.023978 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 32603
I20260810 07:27:38.024121 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:27:38.024446 12919 external_mini_cluster.cc:1428] Setting key 9f8144500106bdfbce5071238a22283b
I20260810 07:27:38.025434 32702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:38.031314 32702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80: Bootstrap starting.
Aug 10 07:27:38 dist-test-slave-ffrd krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:27:38.034152 32702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:38.034981 32702 log.cc:824] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:38.037897 32702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80: No bootstrap required, opened a new log
I20260810 07:27:38.041369 32707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:38.025992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54360 (local address 127.12.157.254:44937)
0810 07:27:38.026470 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:38.026487 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:38.026523 (+ 36us) server_negotiation.cc:407] Connection header received
0810 07:27:38.027185 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:38.027202 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:38.027527 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:38.027892 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:38.028740 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:38.029821 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:38.030722 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:38.031212 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:38.031400 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:38.034563 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:38.034568 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:38.034584 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:38.034614 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:38.037401 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:38.038138 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:38.038147 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:38.038157 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:38.038287 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:38.038930 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:38.038933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:38.038935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:38.039422 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:38.039601 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:38.040066 (+ 465us) server_negotiation.cc:299] Negotiation successful
0810 07:27:38.040487 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":146,"threads_started":1}
I20260810 07:27:38.041999 32702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bc2b56025342ec869b7536820e2c80" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44937 } }
I20260810 07:27:38.042275 32702 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:38.042449 32702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36bc2b56025342ec869b7536820e2c80, State: Initialized, Role: FOLLOWER
I20260810 07:27:38.042999 32702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bc2b56025342ec869b7536820e2c80" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44937 } }
I20260810 07:27:38.043174 32702 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:27:38.043242 32702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:27:38.043340 32702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:38.044528 32702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bc2b56025342ec869b7536820e2c80" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44937 } }
I20260810 07:27:38.044855 32702 leader_election.cc:302] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36bc2b56025342ec869b7536820e2c80; no voters:
I20260810 07:27:38.045256 32702 leader_election.cc:288] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:27:38.045401 32709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:27:38.045744 32709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [term 1 LEADER]: Becoming Leader. State: Replica: 36bc2b56025342ec869b7536820e2c80, State: Running, Role: LEADER
I20260810 07:27:38.046180 32709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bc2b56025342ec869b7536820e2c80" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44937 } }
I20260810 07:27:38.047252 32702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:27:38.048265 32711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36bc2b56025342ec869b7536820e2c80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bc2b56025342ec869b7536820e2c80" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44937 } } }
I20260810 07:27:38.048239 32710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36bc2b56025342ec869b7536820e2c80. Latest consensus state: current_term: 1 leader_uuid: "36bc2b56025342ec869b7536820e2c80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bc2b56025342ec869b7536820e2c80" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44937 } } }
I20260810 07:27:38.048434 32710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:38.048434 32711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:38.049108 32721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:27:38.051968 32721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:27:38.052501 32702 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 07:27:38.062973 32721 catalog_manager.cc:1347] Generated new cluster ID: 206f2ec53572484f81170a08e70d6ceb
I20260810 07:27:38.063118 32721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:27:38.085392 32721 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:27:38.086969 32721 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:27:38.094151 32721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80: Generated new TSK 0
I20260810 07:27:38.095007 32721 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
I20260810 07:27:38.793111 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.157.212:33953/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:0
--local_ip_for_outbound_sockets=127.12.157.193
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:44937
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:38.917131 301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:38.917418 301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:38.917544 301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:38.921968 301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:38.922076 301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:38.922163 301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:27:38.928045 301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.157.212:33953/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:44937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:27:38.929268 301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:38.930328 301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:38.940825 307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:38.940999 315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:38.940837 312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:38.942353 301 server_base.cc:1034] running on GCE node
I20260810 07:27:38.942808 301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:38.943481 301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:38.944675 301 hybrid_clock.cc:648] HybridClock initialized: now 1786346858944658 us; error 33 us; skew 500 ppm
Aug 10 07:27:38 dist-test-slave-ffrd krb5kdc[31961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346858, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:38.948180 301 init.cc:377] Logged in from keytab as kudu/127.12.157.193@KRBTEST.COM (short username kudu)
I20260810 07:27:38.949375 301 webserver.cc:460] Webserver started at http://127.12.157.193:34143/ using document root <none> and password file <none>
I20260810 07:27:38.950040 301 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:38.950140 301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:38.950395 301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 07:27:38 dist-test-slave-ffrd krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346858, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for HTTP/127.12.157.212@KRBTEST.COM
I20260810 07:27:38.967344 301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/data/instance:
uuid: "b7b301973de3499da5d5284c9bae437c"
format_stamp: "Formatted at 2026-08-10 07:27:38 on dist-test-slave-ffrd"
server_key: "a4c5be3cc754b1c68436daa2ed5f7822"
server_key_iv: "54e12683bbc81a9e956a3f8ea1cf0d31"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:38.968499 301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance:
uuid: "b7b301973de3499da5d5284c9bae437c"
format_stamp: "Formatted at 2026-08-10 07:27:38 on dist-test-slave-ffrd"
server_key: "a4c5be3cc754b1c68436daa2ed5f7822"
server_key_iv: "54e12683bbc81a9e956a3f8ea1cf0d31"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:38.974099 301 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 07:27:38.997373 328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:38.999207 301 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.000s
I20260810 07:27:38.999403 301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "b7b301973de3499da5d5284c9bae437c"
format_stamp: "Formatted at 2026-08-10 07:27:38 on dist-test-slave-ffrd"
server_key: "a4c5be3cc754b1c68436daa2ed5f7822"
server_key_iv: "54e12683bbc81a9e956a3f8ea1cf0d31"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:38.999599 301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:39.019455 301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:39.023401 301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:39.023664 301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:39.024593 301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:39.025801 301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:39.025919 301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:39.026050 301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:39.026137 301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:39.040333 301 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:42479
I20260810 07:27:39.040403 509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:42479 every 8 connection(s)
I20260810 07:27:39.041646 301 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:27:39.044206 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 301
I20260810 07:27:39.044341 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance
I20260810 07:27:39.044689 12919 external_mini_cluster.cc:1428] Setting key 8eef9416ed7e9becae1cf088c7755208
WARNING: no policy specified for kudu/127.12.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.194@KRBTEST.COM" created.
Aug 10 07:27:39 dist-test-slave-ffrd krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346858, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:27:39.074544 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:39.047517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:42023 (local address 127.12.157.254:44937)
0810 07:27:39.053194 (+ 5677us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:39.053204 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:39.054920 (+ 1716us) server_negotiation.cc:407] Connection header received
0810 07:27:39.054980 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:39.054983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:39.055038 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:39.055127 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:39.059030 (+ 3903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:39.060473 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:39.062351 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:39.062712 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:39.062804 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:39.069061 (+ 6257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:39.069065 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:39.069068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:39.069098 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:39.071062 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:39.071712 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:39.071716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:39.071720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:39.071814 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:39.073089 (+ 1275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:39.073093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:39.073095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:39.073369 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:39.073466 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:39.074196 (+ 730us) server_negotiation.cc:299] Negotiation successful
0810 07:27:39.074353 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5577,"thread_start_us":102,"threads_started":1}
I20260810 07:27:39.076321 515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:39.046353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:44937 (local address 127.12.157.193:42023)
0810 07:27:39.053357 (+ 7004us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:39.053446 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:39.054742 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:39.057658 (+ 2916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:39.057670 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:39.058081 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:39.058818 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:39.058843 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:39.060673 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:39.060677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:39.062194 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:39.062206 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:39.063066 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:39.063070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:39.063276 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:39.064051 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:39.064074 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:39.068781 (+ 4707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:39.071207 (+ 2426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:39.071213 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:39.071233 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:39.071593 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:39.071943 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:39.071946 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:39.071948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:39.072124 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:39.073545 (+ 1421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:39.073552 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:39.073898 (+ 346us) client_negotiation.cc:769] Sending connection context
0810 07:27:39.074824 (+ 926us) client_negotiation.cc:240] Negotiation successful
0810 07:27:39.075198 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6708,"thread_start_us":137,"threads_started":1}
I20260810 07:27:39.077852 511 heartbeater.cc:347] Connected to a master server at 127.12.157.254:44937
I20260810 07:27:39.078122 511 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:39.078835 511 heartbeater.cc:511] Master 127.12.157.254:44937 requested a full tablet report, sending...
I20260810 07:27:39.080709 32642 ts_manager.cc:194] Registered new tserver with Master: b7b301973de3499da5d5284c9bae437c (127.12.157.193:42479)
I20260810 07:27:39.082329 32642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.193@KRBTEST.COM'} at 127.12.157.193:42023
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
I20260810 07:27:39.092387 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.157.212:33953/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:0
--local_ip_for_outbound_sockets=127.12.157.194
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:44937
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 07:27:39 dist-test-slave-ffrd krb5kdc[31961](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786346859, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 07:27:39.249087 528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:39.249428 528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:39.249547 528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:39.253909 528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:39.253998 528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:39.254105 528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:27:39.259920 528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.157.212:33953/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:44937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:27:39.262156 528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:39.263365 528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:39.275010 541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:39.275257 540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:39.275098 543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:39.275797 528 server_base.cc:1034] running on GCE node
I20260810 07:27:39.276201 528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:39.277026 528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:39.278213 528 hybrid_clock.cc:648] HybridClock initialized: now 1786346859278185 us; error 49 us; skew 500 ppm
Aug 10 07:27:39 dist-test-slave-ffrd krb5kdc[31961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346859, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:39.281250 528 init.cc:377] Logged in from keytab as kudu/127.12.157.194@KRBTEST.COM (short username kudu)
I20260810 07:27:39.282160 528 webserver.cc:460] Webserver started at http://127.12.157.194:35301/ using document root <none> and password file <none>
I20260810 07:27:39.282832 528 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:39.282931 528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:39.283200 528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 07:27:39 dist-test-slave-ffrd krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346859, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for HTTP/127.12.157.212@KRBTEST.COM
I20260810 07:27:39.298041 528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/data/instance:
uuid: "b50dcac920074471a959d6ee8e5eb74b"
format_stamp: "Formatted at 2026-08-10 07:27:39 on dist-test-slave-ffrd"
server_key: "90ca5bd104706dabad3d115c3b98aad2"
server_key_iv: "cb5e8a69ca169dca933e564eaa77723e"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:39.298870 528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance:
uuid: "b50dcac920074471a959d6ee8e5eb74b"
format_stamp: "Formatted at 2026-08-10 07:27:39 on dist-test-slave-ffrd"
server_key: "90ca5bd104706dabad3d115c3b98aad2"
server_key_iv: "cb5e8a69ca169dca933e564eaa77723e"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:39.304481 528 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 07:27:39.328837 552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:39.330794 528 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.003s sys 0.001s
I20260810 07:27:39.331002 528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "b50dcac920074471a959d6ee8e5eb74b"
format_stamp: "Formatted at 2026-08-10 07:27:39 on dist-test-slave-ffrd"
server_key: "90ca5bd104706dabad3d115c3b98aad2"
server_key_iv: "cb5e8a69ca169dca933e564eaa77723e"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:39.331159 528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:39.350453 528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:39.354351 528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:39.354610 528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:39.355396 528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:39.356567 528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:39.356684 528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:39.356787 528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:39.356873 528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:39.367795 528 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:46147
I20260810 07:27:39.369100 528 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:27:39.369748 715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:46147 every 8 connection(s)
I20260810 07:27:39.376181 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 528
I20260810 07:27:39.376423 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance
I20260810 07:27:39.376773 12919 external_mini_cluster.cc:1428] Setting key bae071fb2e5a478187173b7611b280f8
Aug 10 07:27:39 dist-test-slave-ffrd krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346859, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:27:39.393313 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:39.373839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:47359 (local address 127.12.157.254:44937)
0810 07:27:39.373986 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:39.373991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:39.374799 (+ 808us) server_negotiation.cc:407] Connection header received
0810 07:27:39.376091 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:39.376095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:39.376161 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:39.376220 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:39.378129 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:39.378959 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:39.384715 (+ 5756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:39.385116 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:39.385209 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:39.388635 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:39.388640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:39.388643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:39.388674 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:39.390534 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:39.391250 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:39.391254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:39.391256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:39.391366 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:39.391818 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:39.391822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:39.391824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:39.392070 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:39.392168 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:39.392975 (+ 807us) server_negotiation.cc:299] Negotiation successful
0810 07:27:39.393127 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for kudu/127.12.157.195@KRBTEST.COM; defaulting to no policy
I20260810 07:27:39.395397 718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:39.372617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:44937 (local address 127.12.157.194:47359)
0810 07:27:39.374587 (+ 1970us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:39.374649 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:39.375813 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:39.376796 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:39.376806 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:39.377205 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:39.377934 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:39.377956 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:39.379132 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:39.379135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:39.384540 (+ 5405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:39.384556 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:39.385461 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:39.385465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:39.385690 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:39.386468 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:39.386494 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:39.388406 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:39.390697 (+ 2291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:39.390705 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:39.390725 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:39.391115 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:39.391491 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:39.391494 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:39.391496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:39.391701 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:39.392243 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:39.392251 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:39.392613 (+ 362us) client_negotiation.cc:769] Sending connection context
0810 07:27:39.393833 (+ 1220us) client_negotiation.cc:240] Negotiation successful
0810 07:27:39.394231 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1736,"thread_start_us":200,"threads_started":1}
I20260810 07:27:39.396991 716 heartbeater.cc:347] Connected to a master server at 127.12.157.254:44937
I20260810 07:27:39.397262 716 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:39.397930 716 heartbeater.cc:511] Master 127.12.157.254:44937 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.195@KRBTEST.COM" created.
I20260810 07:27:39.403348 32642 ts_manager.cc:194] Registered new tserver with Master: b50dcac920074471a959d6ee8e5eb74b (127.12.157.194:46147)
I20260810 07:27:39.404161 32642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.194@KRBTEST.COM'} at 127.12.157.194:47359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
I20260810 07:27:39.426250 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.157.212:33953/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:0
--local_ip_for_outbound_sockets=127.12.157.195
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:44937
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:39.599613 722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:39.599970 722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:39.600042 722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:39.605988 722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:39.606066 722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:39.606156 722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:27:39.613858 722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.157.212:33953/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:44937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:27:39.615186 722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:39.616504 722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:39.629077 734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:39.629262 729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:39.629083 731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:39.630609 722 server_base.cc:1034] running on GCE node
I20260810 07:27:39.631091 722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:39.631870 722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:39.633016 722 hybrid_clock.cc:648] HybridClock initialized: now 1786346859632991 us; error 38 us; skew 500 ppm
Aug 10 07:27:39 dist-test-slave-ffrd krb5kdc[31961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346859, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:39.636693 722 init.cc:377] Logged in from keytab as kudu/127.12.157.195@KRBTEST.COM (short username kudu)
I20260810 07:27:39.637902 722 webserver.cc:460] Webserver started at http://127.12.157.195:41029/ using document root <none> and password file <none>
I20260810 07:27:39.638640 722 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:39.638737 722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:39.639003 722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 07:27:39 dist-test-slave-ffrd krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346859, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for HTTP/127.12.157.212@KRBTEST.COM
I20260810 07:27:39.667969 722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/data/instance:
uuid: "d7f6d311b26e4b20bb5df7a81fb2a7f4"
format_stamp: "Formatted at 2026-08-10 07:27:39 on dist-test-slave-ffrd"
server_key: "80c792d9fec5c8d6edb342ebe385258b"
server_key_iv: "37d601a2d6cec6e182f4788049814aff"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:39.669018 722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance:
uuid: "d7f6d311b26e4b20bb5df7a81fb2a7f4"
format_stamp: "Formatted at 2026-08-10 07:27:39 on dist-test-slave-ffrd"
server_key: "80c792d9fec5c8d6edb342ebe385258b"
server_key_iv: "37d601a2d6cec6e182f4788049814aff"
server_key_version: "kuduclusterkey@0"
Aug 10 07:27:39 dist-test-slave-ffrd krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346859, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for HTTP/127.12.157.212@KRBTEST.COM
I20260810 07:27:39.674696 722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 07:27:39.691962 748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:39.693619 722 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 07:27:39.693759 722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "d7f6d311b26e4b20bb5df7a81fb2a7f4"
format_stamp: "Formatted at 2026-08-10 07:27:39 on dist-test-slave-ffrd"
server_key: "80c792d9fec5c8d6edb342ebe385258b"
server_key_iv: "37d601a2d6cec6e182f4788049814aff"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:39.693908 722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:39.711905 722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:39.715512 722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:39.715719 722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:39.716378 722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:39.717437 722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:39.717501 722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:39.717562 722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:39.717592 722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:39.730432 722 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:34459
I20260810 07:27:39.731609 722 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:27:39.733062 926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:34459 every 8 connection(s)
I20260810 07:27:39.739228 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 722
I20260810 07:27:39.739377 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance
I20260810 07:27:39.739706 12919 external_mini_cluster.cc:1428] Setting key aaedb8f3d4efe2fcc79968c1c9af0fa1
Aug 10 07:27:39 dist-test-slave-ffrd krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346859, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:27:39.750979 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:39.735278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:48935 (local address 127.12.157.254:44937)
0810 07:27:39.735483 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:39.735488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:39.737238 (+ 1750us) server_negotiation.cc:407] Connection header received
0810 07:27:39.738466 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:39.738470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:39.738526 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:39.738579 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:39.740282 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:39.741157 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:39.742866 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:39.743199 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:39.743289 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:39.746672 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:39.746677 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:39.746681 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:39.746710 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:39.748523 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:39.749185 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:39.749189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:39.749191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:39.749291 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:39.749707 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:39.749710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:39.749712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:39.749930 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:39.750011 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:39.750687 (+ 676us) server_negotiation.cc:299] Negotiation successful
0810 07:27:39.750816 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 07:27:39.752568 932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:39.735826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:44937 (local address 127.12.157.195:48935)
0810 07:27:39.737069 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:39.737124 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:39.738200 (+ 1076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:39.738994 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:39.739003 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:39.739373 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:39.740097 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:39.740115 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:39.741326 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:39.741330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:39.742722 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:39.742734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:39.743496 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:39.743499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:39.743706 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:39.744478 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:39.744500 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:39.746437 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:39.748651 (+ 2214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:39.748657 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:39.748674 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:39.749047 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:39.749424 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:39.749427 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:39.749429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:39.749602 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:39.750078 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:39.750085 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:39.750408 (+ 323us) client_negotiation.cc:769] Sending connection context
0810 07:27:39.751201 (+ 793us) client_negotiation.cc:240] Negotiation successful
0810 07:27:39.751552 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"thread_start_us":167,"threads_started":1}
I20260810 07:27:39.753952 927 heartbeater.cc:347] Connected to a master server at 127.12.157.254:44937
I20260810 07:27:39.754211 927 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:39.754793 927 heartbeater.cc:511] Master 127.12.157.254:44937 requested a full tablet report, sending...
I20260810 07:27:39.756631 32642 ts_manager.cc:194] Registered new tserver with Master: d7f6d311b26e4b20bb5df7a81fb2a7f4 (127.12.157.195:34459)
I20260810 07:27:39.757395 32642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.195@KRBTEST.COM'} at 127.12.157.195:48935
I20260810 07:27:39.767714 12919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:27:39.768627 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 301
I20260810 07:27:39.793104 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 528
I20260810 07:27:39.810367 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 722
I20260810 07:27:39.827708 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 32603
I20260810 07:27:39.846433 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:44937
--webserver_interface=127.12.157.254
--webserver_port=40617
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:44937
--ranger_config_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.157.212:33953/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.157.212:33953/kms with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:39.966514 934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:39.966806 934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:39.966856 934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:39.971221 934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:27:39.971290 934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:39.971310 934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:39.971328 934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:27:39.971342 934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:27:39.976856 934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.157.212:33953/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:44937
--ranger_config_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:44937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=40617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:27:39.977938 934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:39.979596 934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:39.988282 945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:39.988332 943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:39.988282 942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:39.988651 934 server_base.cc:1034] running on GCE node
I20260810 07:27:39.989326 934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:39.990410 934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:39.991570 934 hybrid_clock.cc:648] HybridClock initialized: now 1786346859991552 us; error 31 us; skew 500 ppm
Aug 10 07:27:39 dist-test-slave-ffrd krb5kdc[31961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346859, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:39.994444 934 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:27:39.995270 934 webserver.cc:460] Webserver started at http://127.12.157.254:40617/ using document root <none> and password file <none>
I20260810 07:27:39.995793 934 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:39.995862 934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 07:27:40 dist-test-slave-ffrd krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346859, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for HTTP/127.12.157.212@KRBTEST.COM
I20260810 07:27:40.075392 934 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 07:27:40.078125 954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:40.079640 934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:27:40.079905 934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "36bc2b56025342ec869b7536820e2c80"
format_stamp: "Formatted at 2026-08-10 07:27:37 on dist-test-slave-ffrd"
server_key: "b5ab6e7a2b2c97d1e47a5b09a0080211"
server_key_iv: "d659d38608ceced643678a98d6ff0455"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:40.080504 934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:40.104737 934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:40.107956 934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:40.108191 934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:40.115895 934 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:44937
I20260810 07:27:40.116027 1031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:44937 every 8 connection(s)
I20260810 07:27:40.117107 934 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:27:40.121505 1033 sys_catalog.cc:263] Verifying existing consensus state
I20260810 07:27:40.123155 1033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80: Bootstrap starting.
I20260810 07:27:40.126325 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 934
I20260810 07:27:40.126838 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.157.212:33953/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:42479
--local_ip_for_outbound_sockets=127.12.157.193
--tserver_master_addrs=127.12.157.254:44937
--webserver_port=34143
--webserver_interface=127.12.157.193
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.157.212:33953/kms with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:27:40.141297 1033 log.cc:824] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:40.148752 1033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 07:27:40.149266 1033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80: Bootstrap complete.
I20260810 07:27:40.153666 1033 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bc2b56025342ec869b7536820e2c80" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44937 } }
I20260810 07:27:40.154096 1033 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36bc2b56025342ec869b7536820e2c80, State: Initialized, Role: FOLLOWER
I20260810 07:27:40.154668 1033 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bc2b56025342ec869b7536820e2c80" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44937 } }
I20260810 07:27:40.154790 1033 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:27:40.154878 1033 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:27:40.155005 1033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [term 1 FOLLOWER]: Advancing to term 2
I20260810 07:27:40.157007 1033 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bc2b56025342ec869b7536820e2c80" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44937 } }
I20260810 07:27:40.157362 1033 leader_election.cc:302] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36bc2b56025342ec869b7536820e2c80; no voters:
I20260810 07:27:40.157733 1033 leader_election.cc:288] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 07:27:40.157824 1039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 07:27:40.158059 1039 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [term 2 LEADER]: Becoming Leader. State: Replica: 36bc2b56025342ec869b7536820e2c80, State: Running, Role: LEADER
I20260810 07:27:40.158439 1039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bc2b56025342ec869b7536820e2c80" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44937 } }
I20260810 07:27:40.159034 1033 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:27:40.159889 1040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "36bc2b56025342ec869b7536820e2c80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bc2b56025342ec869b7536820e2c80" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44937 } } }
I20260810 07:27:40.160032 1040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:40.160427 1041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36bc2b56025342ec869b7536820e2c80. Latest consensus state: current_term: 2 leader_uuid: "36bc2b56025342ec869b7536820e2c80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36bc2b56025342ec869b7536820e2c80" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44937 } } }
I20260810 07:27:40.160588 1041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80 [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:40.161054 1049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:27:40.164376 1049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:27:40.164903 1033 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 07:27:40.165148 1049 catalog_manager.cc:1259] Loaded cluster ID: 206f2ec53572484f81170a08e70d6ceb
I20260810 07:27:40.165189 1049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:27:40.170820 1049 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:27:40.171792 1049 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80: Loaded TSK: 0
I20260810 07:27:40.173393 1049 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 07:27:40.253161 1035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:40.253444 1035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:40.253547 1035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:40.258188 1035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:40.258279 1035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:40.258383 1035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:27:40.264408 1035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.157.212:33953/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:42479
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=34143
--tserver_master_addrs=127.12.157.254:44937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:27:40.265573 1035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:40.266620 1035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:40.279592 1088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:40.279402 1085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:40.279402 1090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:40.281123 1035 server_base.cc:1034] running on GCE node
I20260810 07:27:40.281611 1035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:40.282351 1035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:40.283586 1035 hybrid_clock.cc:648] HybridClock initialized: now 1786346860283546 us; error 47 us; skew 500 ppm
Aug 10 07:27:40 dist-test-slave-ffrd krb5kdc[31961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346860, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:40.287451 1035 init.cc:377] Logged in from keytab as kudu/127.12.157.193@KRBTEST.COM (short username kudu)
I20260810 07:27:40.288556 1035 webserver.cc:460] Webserver started at http://127.12.157.193:34143/ using document root <none> and password file <none>
I20260810 07:27:40.289315 1035 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:40.289444 1035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 07:27:40 dist-test-slave-ffrd krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346860, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for HTTP/127.12.157.212@KRBTEST.COM
I20260810 07:27:40.309918 1035 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 07:27:40.314937 1104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:40.316437 1035 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260810 07:27:40.316596 1035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "b7b301973de3499da5d5284c9bae437c"
format_stamp: "Formatted at 2026-08-10 07:27:38 on dist-test-slave-ffrd"
server_key: "a4c5be3cc754b1c68436daa2ed5f7822"
server_key_iv: "54e12683bbc81a9e956a3f8ea1cf0d31"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:40.317283 1035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:40.336293 1035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:40.340190 1035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:40.340492 1035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:40.341409 1035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:40.342489 1035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:40.342568 1035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:40.342664 1035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:40.342722 1035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:40.355268 1035 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:42479
I20260810 07:27:40.355288 1265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:42479 every 8 connection(s)
I20260810 07:27:40.356463 1035 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:27:40.358290 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 1035
I20260810 07:27:40.358775 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.157.212:33953/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:46147
--local_ip_for_outbound_sockets=127.12.157.194
--tserver_master_addrs=127.12.157.254:44937
--webserver_port=35301
--webserver_interface=127.12.157.194
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.157.212:33953/kms with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 07:27:40 dist-test-slave-ffrd krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346860, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:27:40.394404 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:40.361521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:35871 (local address 127.12.157.254:44937)
0810 07:27:40.363457 (+ 1936us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:40.363482 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:40.363534 (+ 52us) server_negotiation.cc:407] Connection header received
0810 07:27:40.368505 (+ 4971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:40.368523 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:40.369007 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:40.369493 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:40.371336 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:40.372911 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:40.374572 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:40.375323 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:40.375504 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:40.388221 (+ 12717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:40.388228 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:40.388246 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:40.388280 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:40.390360 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:40.391010 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:40.391016 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:40.391021 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:40.391150 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:40.391513 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:40.391516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:40.391518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:40.391990 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:40.392101 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:40.393123 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0810 07:27:40.393417 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1583,"thread_start_us":142,"threads_started":1}
I20260810 07:27:40.395498 1269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:40.359242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:44937 (local address 127.12.157.193:35871)
0810 07:27:40.362648 (+ 3406us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:40.362706 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:40.363889 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:40.370077 (+ 6188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:40.370089 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:40.370495 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:40.371123 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:40.371142 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:40.373092 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:40.373096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:40.374420 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:40.374433 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:40.385126 (+ 10693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:40.385133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:40.385389 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:40.386167 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:40.386188 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:40.387949 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:40.390510 (+ 2561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:40.390517 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:40.390532 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:40.390870 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:40.391251 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:40.391254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:40.391256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:40.391411 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:40.392163 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:40.392169 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:40.392453 (+ 284us) client_negotiation.cc:769] Sending connection context
0810 07:27:40.392694 (+ 241us) client_negotiation.cc:240] Negotiation successful
0810 07:27:40.394727 (+ 2033us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3152,"thread_start_us":154,"threads_started":1}
I20260810 07:27:40.399678 1266 heartbeater.cc:347] Connected to a master server at 127.12.157.254:44937
I20260810 07:27:40.399967 1266 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:40.400592 1266 heartbeater.cc:511] Master 127.12.157.254:44937 requested a full tablet report, sending...
W20260810 07:27:40.402972 1266 heartbeater.cc:644] Failed to heartbeat to 127.12.157.254:44937 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 07:27:40.403098 1266 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:40.403290 1266 heartbeater.cc:511] Master 127.12.157.254:44937 requested a full tablet report, sending...
I20260810 07:27:40.404217 1266 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:40.404403 1266 heartbeater.cc:511] Master 127.12.157.254:44937 requested a full tablet report, sending...
W20260810 07:27:40.405269 1266 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 07:27:40.533617 1270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:40.534075 1270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:40.534183 1270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:40.538924 1270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:40.539012 1270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:40.539125 1270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:27:40.546721 1270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.157.212:33953/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:46147
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=35301
--tserver_master_addrs=127.12.157.254:44937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:27:40.548317 1270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:40.549752 1270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:40.560052 1278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:40.560077 1282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:40.560103 1277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:40.560567 1270 server_base.cc:1034] running on GCE node
I20260810 07:27:40.561023 1270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:40.561766 1270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:40.562978 1270 hybrid_clock.cc:648] HybridClock initialized: now 1786346860562906 us; error 87 us; skew 500 ppm
Aug 10 07:27:40 dist-test-slave-ffrd krb5kdc[31961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346860, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:40.566248 1270 init.cc:377] Logged in from keytab as kudu/127.12.157.194@KRBTEST.COM (short username kudu)
I20260810 07:27:40.567312 1270 webserver.cc:460] Webserver started at http://127.12.157.194:35301/ using document root <none> and password file <none>
I20260810 07:27:40.567926 1270 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:40.568089 1270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 07:27:40 dist-test-slave-ffrd krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346860, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for HTTP/127.12.157.212@KRBTEST.COM
I20260810 07:27:40.602378 1270 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260810 07:27:40.605509 1295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:40.606997 1270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 07:27:40.607144 1270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "b50dcac920074471a959d6ee8e5eb74b"
format_stamp: "Formatted at 2026-08-10 07:27:39 on dist-test-slave-ffrd"
server_key: "90ca5bd104706dabad3d115c3b98aad2"
server_key_iv: "cb5e8a69ca169dca933e564eaa77723e"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:40.607862 1270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:40.624748 1270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:40.629387 1270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:40.629680 1270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:40.630504 1270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:40.631517 1270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:40.631569 1270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:40.631613 1270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:40.631628 1270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:40.643494 1270 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:46147
I20260810 07:27:40.644865 1270 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:27:40.647621 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 1270
I20260810 07:27:40.648120 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.157.212:33953/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:34459
--local_ip_for_outbound_sockets=127.12.157.195
--tserver_master_addrs=127.12.157.254:44937
--webserver_port=41029
--webserver_interface=127.12.157.195
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:37913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.157.212:33953/kms with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:27:40.652693 1464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:46147 every 8 connection(s)
Aug 10 07:27:40 dist-test-slave-ffrd krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346860, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:27:40.671674 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:40.647240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:59365 (local address 127.12.157.254:44937)
0810 07:27:40.647405 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:40.647410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:40.653710 (+ 6300us) server_negotiation.cc:407] Connection header received
0810 07:27:40.655018 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:40.655022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:40.655097 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:40.655156 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:40.657041 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:40.658211 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:40.660201 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:40.660598 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:40.660684 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:40.666397 (+ 5713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:40.666401 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:40.666404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:40.666436 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:40.668333 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:40.669070 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:40.669074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:40.669076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:40.669169 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:40.670173 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:40.670177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:40.670178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:40.670455 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:40.670546 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:40.671321 (+ 775us) server_negotiation.cc:299] Negotiation successful
0810 07:27:40.671489 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 07:27:40.673659 1470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:40.647766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:44937 (local address 127.12.157.194:59365)
0810 07:27:40.653478 (+ 5712us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:40.653545 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:40.654711 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:40.655640 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:40.655651 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:40.656050 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:40.656780 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:40.656800 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:40.658445 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:40.658449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:40.660041 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:40.660054 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:40.661394 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:40.661398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:40.661876 (+ 478us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:40.664311 (+ 2435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:40.664334 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:40.666131 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:40.668488 (+ 2357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:40.668496 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:40.668515 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:40.668919 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:40.669761 (+ 842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:40.669765 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:40.669768 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:40.670016 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:40.670623 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:40.670630 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:40.671003 (+ 373us) client_negotiation.cc:769] Sending connection context
0810 07:27:40.671948 (+ 945us) client_negotiation.cc:240] Negotiation successful
0810 07:27:40.672350 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5419,"thread_start_us":153,"threads_started":1}
I20260810 07:27:40.675056 1465 heartbeater.cc:347] Connected to a master server at 127.12.157.254:44937
I20260810 07:27:40.675321 1465 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:40.675988 1465 heartbeater.cc:511] Master 127.12.157.254:44937 requested a full tablet report, sending...
W20260810 07:27:40.678053 1465 heartbeater.cc:644] Failed to heartbeat to 127.12.157.254:44937 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 07:27:40.678187 1465 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:40.678393 1465 heartbeater.cc:511] Master 127.12.157.254:44937 requested a full tablet report, sending...
I20260810 07:27:40.679262 1465 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:40.679580 1465 heartbeater.cc:511] Master 127.12.157.254:44937 requested a full tablet report, sending...
W20260810 07:27:40.681586 1465 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 07:27:40.804024 1471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:40.804304 1471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:40.804348 1471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:40.809360 1471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:40.809429 1471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:40.809520 1471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:27:40.815629 1471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:37913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.157.212:33953/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:34459
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=41029
--tserver_master_addrs=127.12.157.254:44937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:27:40.816823 1471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:40.817971 1471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:40.829282 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:40.829799 1482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:40.829959 1471 server_base.cc:1034] running on GCE node
W20260810 07:27:40.837615 1486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:40.838251 1471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:40.839272 1471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:40.840500 1471 hybrid_clock.cc:648] HybridClock initialized: now 1786346860840465 us; error 54 us; skew 500 ppm
Aug 10 07:27:40 dist-test-slave-ffrd krb5kdc[31961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346860, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:40.844408 1471 init.cc:377] Logged in from keytab as kudu/127.12.157.195@KRBTEST.COM (short username kudu)
I20260810 07:27:40.845779 1471 webserver.cc:460] Webserver started at http://127.12.157.195:41029/ using document root <none> and password file <none>
I20260810 07:27:40.846602 1471 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:40.846791 1471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 07:27:40 dist-test-slave-ffrd krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346860, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for HTTP/127.12.157.212@KRBTEST.COM
I20260810 07:27:40.868759 1471 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 07:27:40.872048 1495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:40.873538 1471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:27:40.873649 1471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "d7f6d311b26e4b20bb5df7a81fb2a7f4"
format_stamp: "Formatted at 2026-08-10 07:27:39 on dist-test-slave-ffrd"
server_key: "80c792d9fec5c8d6edb342ebe385258b"
server_key_iv: "37d601a2d6cec6e182f4788049814aff"
server_key_version: "kuduclusterkey@0"
I20260810 07:27:40.874328 1471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:40.893476 1471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:40.896903 1471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:40.897156 1471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:40.897908 1471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:40.898978 1471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:40.899044 1471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:40.899111 1471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:40.899161 1471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:40.911397 1471 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:34459
I20260810 07:27:40.912711 1471 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:27:40.913147 1685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:34459 every 8 connection(s)
I20260810 07:27:40.922737 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 1471
Aug 10 07:27:40 dist-test-slave-ffrd krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346860, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:27:40.930011 1271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:40.915606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:38805 (local address 127.12.157.254:44937)
0810 07:27:40.915738 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:40.915743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:40.916671 (+ 928us) server_negotiation.cc:407] Connection header received
0810 07:27:40.917727 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:40.917731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:40.917796 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:40.917911 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:40.919206 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:40.919989 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:40.921333 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:40.921673 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:40.921757 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:40.925577 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:40.925582 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:40.925584 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:40.925615 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:40.927457 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:40.928090 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:40.928094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:40.928096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:40.928187 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:40.928633 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:40.928637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:40.928639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:40.928884 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:40.929004 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:40.929702 (+ 698us) server_negotiation.cc:299] Negotiation successful
0810 07:27:40.929848 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 07:27:40.931618 1690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:40.916049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:44937 (local address 127.12.157.195:38805)
0810 07:27:40.916519 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:40.916571 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:40.917491 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:40.918064 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:40.918070 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:40.918335 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:40.919019 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:40.919040 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:40.920131 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:40.920134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:40.921184 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:40.921195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:40.921980 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:40.921984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:40.922174 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:40.923233 (+ 1059us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:40.923253 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:40.925379 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:40.927594 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:40.927600 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:40.927615 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:40.927964 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:40.928305 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:40.928308 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:40.928310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:40.928512 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:40.929078 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:40.929084 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:40.929402 (+ 318us) client_negotiation.cc:769] Sending connection context
0810 07:27:40.930247 (+ 845us) client_negotiation.cc:240] Negotiation successful
0810 07:27:40.930583 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":140,"threads_started":1}
I20260810 07:27:40.932992 1687 heartbeater.cc:347] Connected to a master server at 127.12.157.254:44937
I20260810 07:27:40.933238 1687 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:40.933821 1687 heartbeater.cc:511] Master 127.12.157.254:44937 requested a full tablet report, sending...
W20260810 07:27:40.935635 1687 heartbeater.cc:644] Failed to heartbeat to 127.12.157.254:44937 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 07:27:40.935771 1687 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:40.935950 1687 heartbeater.cc:511] Master 127.12.157.254:44937 requested a full tablet report, sending...
I20260810 07:27:40.936810 1687 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:40.937031 1687 heartbeater.cc:511] Master 127.12.157.254:44937 requested a full tablet report, sending...
W20260810 07:27:40.937883 1687 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 07:27:40.938800 1693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:40.923451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54366 (local address 127.12.157.254:44937)
0810 07:27:40.928950 (+ 5499us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:40.928955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:40.928997 (+ 42us) server_negotiation.cc:407] Connection header received
0810 07:27:40.929034 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:40.929037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:40.929083 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:40.929169 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:40.930190 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:40.931023 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:40.932303 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:40.932671 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:40.932797 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:40.934655 (+ 1858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:40.934659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:40.934661 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:40.934687 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:40.936423 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:40.937228 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:40.937232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:40.937234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:40.937348 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:40.937831 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:40.937835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:40.937836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:40.938047 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:40.938250 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:40.938481 (+ 231us) server_negotiation.cc:299] Negotiation successful
0810 07:27:40.938616 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5419,"thread_start_us":92,"threads_started":1}
Aug 10 07:27:41 dist-test-slave-ffrd krb5kdc[31961](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786346861, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:41.405535 1266 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:41.405874 1266 heartbeater.cc:511] Master 127.12.157.254:44937 requested a full tablet report, sending...
I20260810 07:27:41.407279 978 ts_manager.cc:194] Registered new tserver with Master: b7b301973de3499da5d5284c9bae437c (127.12.157.193:42479)
I20260810 07:27:41.409009 978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.193@KRBTEST.COM'} at 127.12.157.193:35871
I20260810 07:27:41.681808 1465 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:41.682148 1465 heartbeater.cc:511] Master 127.12.157.254:44937 requested a full tablet report, sending...
I20260810 07:27:41.683094 977 ts_manager.cc:194] Registered new tserver with Master: b50dcac920074471a959d6ee8e5eb74b (127.12.157.194:46147)
I20260810 07:27:41.683955 977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.194@KRBTEST.COM'} at 127.12.157.194:59365
Aug 10 07:27:41 dist-test-slave-ffrd krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346861, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for HTTP/127.12.157.212@KRBTEST.COM
I20260810 07:27:41.938206 1687 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:41.938550 1687 heartbeater.cc:511] Master 127.12.157.254:44937 requested a full tablet report, sending...
I20260810 07:27:41.939519 977 ts_manager.cc:194] Registered new tserver with Master: d7f6d311b26e4b20bb5df7a81fb2a7f4 (127.12.157.195:34459)
I20260810 07:27:41.940409 977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.195@KRBTEST.COM'} at 127.12.157.195:38805
I20260810 07:27:41.944612 12919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:27:41.957474 1719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:41.946568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54374 (local address 127.12.157.254:44937)
0810 07:27:41.947340 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:41.947344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:41.947358 (+ 14us) server_negotiation.cc:407] Connection header received
0810 07:27:41.947398 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:41.947401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:41.947590 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:41.947715 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:41.948696 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:41.949572 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:41.950746 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:41.951103 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:41.951212 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:41.953127 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:41.953131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:41.953134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:41.953163 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:41.955000 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:41.955517 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:41.955521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:41.955523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:41.955627 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:41.955980 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:41.955983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:41.955985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:41.956215 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:41.956344 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:41.957158 (+ 814us) server_negotiation.cc:299] Negotiation successful
0810 07:27:41.957292 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":100,"threads_started":1}
I20260810 07:27:41.961925 977 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:27:41.965751 977 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Aug 10 07:27:41 dist-test-slave-ffrd krb5kdc[31961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346861, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for HTTP/127.12.157.212@KRBTEST.COM
I20260810 07:27:41.990177 1729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:41.978395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:34459 (local address 127.0.0.1:57754)
0810 07:27:41.982418 (+ 4023us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:41.982471 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:41.982603 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:41.983349 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:41.983355 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:41.983377 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:41.983508 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:41.983515 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:41.985071 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:41.985075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:41.987309 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:41.987321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:41.988691 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:41.988695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:41.988810 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:41.988853 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 07:27:41.989864 (+ 1011us) client_negotiation.cc:240] Negotiation successful
0810 07:27:41.990009 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3848,"thread_start_us":100,"threads_started":1}
I20260810 07:27:41.990644 1732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:41.979269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57754 (local address 127.12.157.195:34459)
0810 07:27:41.982167 (+ 2898us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:41.982173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:41.982735 (+ 562us) server_negotiation.cc:407] Connection header received
0810 07:27:41.982827 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:41.982833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:41.983047 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:41.983182 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:41.983634 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:41.984905 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:41.987470 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:41.988347 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:41.988477 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:41.988620 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:41.990376 (+ 1756us) server_negotiation.cc:299] Negotiation successful
0810 07:27:41.990501 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2799,"thread_start_us":85,"threads_started":1}
I20260810 07:27:41.994935 1735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:41.981552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:46147 (local address 127.0.0.1:53254)
0810 07:27:41.989470 (+ 7918us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:41.989482 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:41.989558 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:41.990546 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:41.990550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:41.990567 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:41.990648 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:41.990654 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:41.991985 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:41.991988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:41.993261 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:41.993273 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:41.994514 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:41.994518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:41.994613 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:41.994637 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 07:27:41.994713 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 07:27:41.994793 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7856,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260810 07:27:41.994562 1568 tablet_service.cc:1467] Processing CreateTablet for tablet dadfc625268741828fef54fdc41e9221 (DEFAULT_TABLE table=test-table [id=7ffe334c13124018916c6bf6f42f04f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:27:41.995362 1737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:41.981719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53254 (local address 127.12.157.194:46147)
0810 07:27:41.989824 (+ 8105us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:41.989832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:41.989858 (+ 26us) server_negotiation.cc:407] Connection header received
0810 07:27:41.989945 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:41.989951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:41.990185 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:41.990374 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:41.990770 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:41.991856 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:41.993412 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:41.994195 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:41.994318 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:41.994444 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:41.995122 (+ 678us) server_negotiation.cc:299] Negotiation successful
0810 07:27:41.995226 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7979,"thread_start_us":103,"threads_started":1}
I20260810 07:27:41.995754 1568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dadfc625268741828fef54fdc41e9221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:41.997220 1733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:41.980358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:42479 (local address 127.0.0.1:50642)
0810 07:27:41.989132 (+ 8774us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:41.989152 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:41.989263 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:41.992882 (+ 3619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:41.992885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:41.992903 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:41.993008 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:41.993014 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:41.994348 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:41.994352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:41.995611 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:41.995621 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:41.996822 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:41.996826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:41.996911 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:41.996934 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 07:27:41.997026 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 07:27:41.997096 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8694,"thread_start_us":646,"threads_started":1}
I20260810 07:27:41.998049 1734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:41.980731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50642 (local address 127.12.157.193:42479)
0810 07:27:41.981987 (+ 1256us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:41.981994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:41.992276 (+ 10282us) server_negotiation.cc:407] Connection header received
0810 07:27:41.992369 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:41.992374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:41.992561 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:41.992701 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:41.993144 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:41.994215 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:41.995752 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:41.996517 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:41.996633 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:41.996749 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:41.997810 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0810 07:27:41.997916 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1141,"thread_start_us":118,"threads_started":1}
I20260810 07:27:41.998361 1368 tablet_service.cc:1467] Processing CreateTablet for tablet dadfc625268741828fef54fdc41e9221 (DEFAULT_TABLE table=test-table [id=7ffe334c13124018916c6bf6f42f04f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:27:41.999452 1368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dadfc625268741828fef54fdc41e9221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:42.000689 1172 tablet_service.cc:1467] Processing CreateTablet for tablet dadfc625268741828fef54fdc41e9221 (DEFAULT_TABLE table=test-table [id=7ffe334c13124018916c6bf6f42f04f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:27:42.001724 1172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dadfc625268741828fef54fdc41e9221. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:42.009690 1739 tablet_bootstrap.cc:492] T dadfc625268741828fef54fdc41e9221 P b7b301973de3499da5d5284c9bae437c: Bootstrap starting.
I20260810 07:27:42.012279 1739 tablet_bootstrap.cc:654] T dadfc625268741828fef54fdc41e9221 P b7b301973de3499da5d5284c9bae437c: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:42.013262 1739 log.cc:824] T dadfc625268741828fef54fdc41e9221 P b7b301973de3499da5d5284c9bae437c: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:42.013361 1738 tablet_bootstrap.cc:492] T dadfc625268741828fef54fdc41e9221 P b50dcac920074471a959d6ee8e5eb74b: Bootstrap starting.
I20260810 07:27:42.015784 1738 tablet_bootstrap.cc:654] T dadfc625268741828fef54fdc41e9221 P b50dcac920074471a959d6ee8e5eb74b: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:42.016651 1738 log.cc:824] T dadfc625268741828fef54fdc41e9221 P b50dcac920074471a959d6ee8e5eb74b: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:42.019543 1742 tablet_bootstrap.cc:492] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4: Bootstrap starting.
I20260810 07:27:42.021484 1739 tablet_bootstrap.cc:492] T dadfc625268741828fef54fdc41e9221 P b7b301973de3499da5d5284c9bae437c: No bootstrap required, opened a new log
I20260810 07:27:42.021751 1739 ts_tablet_manager.cc:1397] T dadfc625268741828fef54fdc41e9221 P b7b301973de3499da5d5284c9bae437c: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260810 07:27:42.022552 1742 tablet_bootstrap.cc:654] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:42.023607 1742 log.cc:824] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:42.026563 1742 tablet_bootstrap.cc:492] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4: No bootstrap required, opened a new log
I20260810 07:27:42.026790 1742 ts_tablet_manager.cc:1397] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 07:27:42.029593 1738 tablet_bootstrap.cc:492] T dadfc625268741828fef54fdc41e9221 P b50dcac920074471a959d6ee8e5eb74b: No bootstrap required, opened a new log
I20260810 07:27:42.029824 1738 ts_tablet_manager.cc:1397] T dadfc625268741828fef54fdc41e9221 P b50dcac920074471a959d6ee8e5eb74b: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20260810 07:27:42.030619 1742 raft_consensus.cc:348] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f6d311b26e4b20bb5df7a81fb2a7f4" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34459 } } peers { permanent_uuid: "b7b301973de3499da5d5284c9bae437c" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42479 } } peers { permanent_uuid: "b50dcac920074471a959d6ee8e5eb74b" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46147 } }
I20260810 07:27:42.030920 1742 raft_consensus.cc:374] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:42.030972 1742 raft_consensus.cc:729] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f6d311b26e4b20bb5df7a81fb2a7f4, State: Initialized, Role: FOLLOWER
I20260810 07:27:42.031685 1742 consensus_queue.cc:260] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f6d311b26e4b20bb5df7a81fb2a7f4" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34459 } } peers { permanent_uuid: "b7b301973de3499da5d5284c9bae437c" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42479 } } peers { permanent_uuid: "b50dcac920074471a959d6ee8e5eb74b" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46147 } }
I20260810 07:27:42.033056 1742 ts_tablet_manager.cc:1428] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 07:27:42.033476 1687 heartbeater.cc:503] Master 127.12.157.254:44937 was elected leader, sending a full tablet report...
I20260810 07:27:42.034220 1738 raft_consensus.cc:348] T dadfc625268741828fef54fdc41e9221 P b50dcac920074471a959d6ee8e5eb74b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f6d311b26e4b20bb5df7a81fb2a7f4" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34459 } } peers { permanent_uuid: "b7b301973de3499da5d5284c9bae437c" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42479 } } peers { permanent_uuid: "b50dcac920074471a959d6ee8e5eb74b" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46147 } }
I20260810 07:27:42.034487 1738 raft_consensus.cc:374] T dadfc625268741828fef54fdc41e9221 P b50dcac920074471a959d6ee8e5eb74b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:42.034534 1738 raft_consensus.cc:729] T dadfc625268741828fef54fdc41e9221 P b50dcac920074471a959d6ee8e5eb74b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b50dcac920074471a959d6ee8e5eb74b, State: Initialized, Role: FOLLOWER
I20260810 07:27:42.035483 1739 raft_consensus.cc:348] T dadfc625268741828fef54fdc41e9221 P b7b301973de3499da5d5284c9bae437c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f6d311b26e4b20bb5df7a81fb2a7f4" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34459 } } peers { permanent_uuid: "b7b301973de3499da5d5284c9bae437c" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42479 } } peers { permanent_uuid: "b50dcac920074471a959d6ee8e5eb74b" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46147 } }
I20260810 07:27:42.035785 1739 raft_consensus.cc:374] T dadfc625268741828fef54fdc41e9221 P b7b301973de3499da5d5284c9bae437c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:42.035897 1739 raft_consensus.cc:729] T dadfc625268741828fef54fdc41e9221 P b7b301973de3499da5d5284c9bae437c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7b301973de3499da5d5284c9bae437c, State: Initialized, Role: FOLLOWER
I20260810 07:27:42.036404 1739 consensus_queue.cc:260] T dadfc625268741828fef54fdc41e9221 P b7b301973de3499da5d5284c9bae437c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f6d311b26e4b20bb5df7a81fb2a7f4" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34459 } } peers { permanent_uuid: "b7b301973de3499da5d5284c9bae437c" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42479 } } peers { permanent_uuid: "b50dcac920074471a959d6ee8e5eb74b" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46147 } }
I20260810 07:27:42.037525 1739 ts_tablet_manager.cc:1428] T dadfc625268741828fef54fdc41e9221 P b7b301973de3499da5d5284c9bae437c: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260810 07:27:42.037966 1266 heartbeater.cc:503] Master 127.12.157.254:44937 was elected leader, sending a full tablet report...
I20260810 07:27:42.038172 1738 consensus_queue.cc:260] T dadfc625268741828fef54fdc41e9221 P b50dcac920074471a959d6ee8e5eb74b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f6d311b26e4b20bb5df7a81fb2a7f4" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34459 } } peers { permanent_uuid: "b7b301973de3499da5d5284c9bae437c" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42479 } } peers { permanent_uuid: "b50dcac920074471a959d6ee8e5eb74b" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46147 } }
I20260810 07:27:42.040413 1738 ts_tablet_manager.cc:1428] T dadfc625268741828fef54fdc41e9221 P b50dcac920074471a959d6ee8e5eb74b: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260810 07:27:42.040794 1465 heartbeater.cc:503] Master 127.12.157.254:44937 was elected leader, sending a full tablet report...
W20260810 07:27:42.121006 1268 tablet.cc:2369] T dadfc625268741828fef54fdc41e9221 P b7b301973de3499da5d5284c9bae437c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:27:42.131028 1745 raft_consensus.cc:482] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:27:42.131309 1745 raft_consensus.cc:504] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f6d311b26e4b20bb5df7a81fb2a7f4" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34459 } } peers { permanent_uuid: "b7b301973de3499da5d5284c9bae437c" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42479 } } peers { permanent_uuid: "b50dcac920074471a959d6ee8e5eb74b" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46147 } }
I20260810 07:27:42.132843 1745 leader_election.cc:288] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7b301973de3499da5d5284c9bae437c (127.12.157.193:42479), b50dcac920074471a959d6ee8e5eb74b (127.12.157.194:46147)
I20260810 07:27:42.139781 1734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:42.134360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:48717 (local address 127.12.157.193:42479)
0810 07:27:42.134484 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:42.134490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:42.134921 (+ 431us) server_negotiation.cc:407] Connection header received
0810 07:27:42.135166 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:42.135170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:42.135229 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:42.135299 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:42.135800 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:42.136608 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:42.138352 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:42.138994 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:42.139095 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:42.139157 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:42.139539 (+ 382us) server_negotiation.cc:299] Negotiation successful
0810 07:27:42.139631 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 07:27:42.140568 1749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:42.133370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:42479 (local address 127.12.157.195:48717)
0810 07:27:42.134820 (+ 1450us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:42.134837 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:42.135017 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:42.135466 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:42.135469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:42.135491 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:42.135650 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:42.135659 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:42.136769 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:42.136773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:42.138174 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:42.138186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:42.139270 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:42.139274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:42.139385 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:42.139410 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 07:27:42.140321 (+ 911us) client_negotiation.cc:240] Negotiation successful
0810 07:27:42.140411 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1376,"thread_start_us":147,"threads_started":1}
I20260810 07:27:42.141551 1199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dadfc625268741828fef54fdc41e9221" candidate_uuid: "d7f6d311b26e4b20bb5df7a81fb2a7f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7b301973de3499da5d5284c9bae437c" is_pre_election: true
I20260810 07:27:42.141906 1199 raft_consensus.cc:2454] T dadfc625268741828fef54fdc41e9221 P b7b301973de3499da5d5284c9bae437c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7f6d311b26e4b20bb5df7a81fb2a7f4 in term 0.
I20260810 07:27:42.143219 1501 leader_election.cc:302] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7b301973de3499da5d5284c9bae437c, d7f6d311b26e4b20bb5df7a81fb2a7f4; no voters:
I20260810 07:27:42.143617 1745 raft_consensus.cc:2781] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:27:42.143708 1745 raft_consensus.cc:482] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:27:42.143755 1745 raft_consensus.cc:3037] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:42.145478 1745 raft_consensus.cc:504] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f6d311b26e4b20bb5df7a81fb2a7f4" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34459 } } peers { permanent_uuid: "b7b301973de3499da5d5284c9bae437c" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42479 } } peers { permanent_uuid: "b50dcac920074471a959d6ee8e5eb74b" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46147 } }
I20260810 07:27:42.145879 1745 leader_election.cc:288] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4 [CANDIDATE]: Term 1 election: Requested vote from peers b7b301973de3499da5d5284c9bae437c (127.12.157.193:42479), b50dcac920074471a959d6ee8e5eb74b (127.12.157.194:46147)
I20260810 07:27:42.146432 1199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dadfc625268741828fef54fdc41e9221" candidate_uuid: "d7f6d311b26e4b20bb5df7a81fb2a7f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7b301973de3499da5d5284c9bae437c"
I20260810 07:27:42.146579 1199 raft_consensus.cc:3037] T dadfc625268741828fef54fdc41e9221 P b7b301973de3499da5d5284c9bae437c [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:42.147953 1199 raft_consensus.cc:2454] T dadfc625268741828fef54fdc41e9221 P b7b301973de3499da5d5284c9bae437c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7f6d311b26e4b20bb5df7a81fb2a7f4 in term 1.
I20260810 07:27:42.148370 1501 leader_election.cc:302] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7b301973de3499da5d5284c9bae437c, d7f6d311b26e4b20bb5df7a81fb2a7f4; no voters:
I20260810 07:27:42.148597 1745 raft_consensus.cc:2781] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:27:42.148837 1745 raft_consensus.cc:686] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4 [term 1 LEADER]: Becoming Leader. State: Replica: d7f6d311b26e4b20bb5df7a81fb2a7f4, State: Running, Role: LEADER
I20260810 07:27:42.149290 1745 consensus_queue.cc:237] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f6d311b26e4b20bb5df7a81fb2a7f4" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34459 } } peers { permanent_uuid: "b7b301973de3499da5d5284c9bae437c" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42479 } } peers { permanent_uuid: "b50dcac920074471a959d6ee8e5eb74b" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46147 } }
I20260810 07:27:42.152889 977 catalog_manager.cc:5495] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7f6d311b26e4b20bb5df7a81fb2a7f4 (127.12.157.195). New cstate: current_term: 1 leader_uuid: "d7f6d311b26e4b20bb5df7a81fb2a7f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7f6d311b26e4b20bb5df7a81fb2a7f4" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7b301973de3499da5d5284c9bae437c" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b50dcac920074471a959d6ee8e5eb74b" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46147 } health_report { overall_health: UNKNOWN } } }
I20260810 07:27:42.154146 1737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:42.134519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:58691 (local address 127.12.157.194:46147)
0810 07:27:42.134643 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:42.134647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:42.141379 (+ 6732us) server_negotiation.cc:407] Connection header received
0810 07:27:42.141588 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:42.141591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:42.141645 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:42.141695 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:42.142091 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:42.142834 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:42.152714 (+ 9880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:42.153391 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:42.153496 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:42.153552 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:42.153909 (+ 357us) server_negotiation.cc:299] Negotiation successful
0810 07:27:42.154002 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 07:27:42.154251 1466 tablet.cc:2369] T dadfc625268741828fef54fdc41e9221 P b50dcac920074471a959d6ee8e5eb74b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:27:42.154630 1749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:42.133841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:46147 (local address 127.12.157.195:58691)
0810 07:27:42.141297 (+ 7456us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:42.141313 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:42.141461 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:42.141845 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:42.141848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:42.141868 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:42.141948 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:42.141957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:42.151487 (+ 9530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:42.151490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:42.152531 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:42.152544 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:42.153656 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:42.153659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:42.153745 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:42.153772 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 07:27:42.154419 (+ 647us) client_negotiation.cc:240] Negotiation successful
0810 07:27:42.154507 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7378,"thread_start_us":97,"threads_started":1}
I20260810 07:27:42.155501 1401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dadfc625268741828fef54fdc41e9221" candidate_uuid: "d7f6d311b26e4b20bb5df7a81fb2a7f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b50dcac920074471a959d6ee8e5eb74b" is_pre_election: true
I20260810 07:27:42.155829 1401 raft_consensus.cc:2454] T dadfc625268741828fef54fdc41e9221 P b50dcac920074471a959d6ee8e5eb74b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7f6d311b26e4b20bb5df7a81fb2a7f4 in term 0.
I20260810 07:27:42.156189 1400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dadfc625268741828fef54fdc41e9221" candidate_uuid: "d7f6d311b26e4b20bb5df7a81fb2a7f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b50dcac920074471a959d6ee8e5eb74b"
I20260810 07:27:42.156327 1400 raft_consensus.cc:3037] T dadfc625268741828fef54fdc41e9221 P b50dcac920074471a959d6ee8e5eb74b [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:42.157806 1400 raft_consensus.cc:2454] T dadfc625268741828fef54fdc41e9221 P b50dcac920074471a959d6ee8e5eb74b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7f6d311b26e4b20bb5df7a81fb2a7f4 in term 1.
W20260810 07:27:42.165052 1689 tablet.cc:2369] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:27:42.194743 1732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:42.189477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57766 (local address 127.12.157.195:34459)
0810 07:27:42.189611 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:42.189616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:42.189630 (+ 14us) server_negotiation.cc:407] Connection header received
0810 07:27:42.189721 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:42.189724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:42.189792 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:42.189905 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:27:42.190342 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:42.191113 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:42.192353 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:42.192673 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:42.192808 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:42.193338 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:27:42.193937 (+ 599us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:27:42.194058 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:42.194531 (+ 473us) server_negotiation.cc:299] Negotiation successful
0810 07:27:42.194597 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 07:27:42.202780 1400 raft_consensus.cc:1261] T dadfc625268741828fef54fdc41e9221 P b50dcac920074471a959d6ee8e5eb74b [term 1 FOLLOWER]: Refusing update from remote peer d7f6d311b26e4b20bb5df7a81fb2a7f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:27:42.203752 1752 consensus_queue.cc:1035] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b50dcac920074471a959d6ee8e5eb74b" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 46147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:27:42.209738 1199 raft_consensus.cc:1261] T dadfc625268741828fef54fdc41e9221 P b7b301973de3499da5d5284c9bae437c [term 1 FOLLOWER]: Refusing update from remote peer d7f6d311b26e4b20bb5df7a81fb2a7f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:27:42.210602 1752 consensus_queue.cc:1035] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7b301973de3499da5d5284c9bae437c" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 42479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:27:42.220193 1756 mvcc.cc:205] Tried to move back new op lower bound from 7316876747573919744 to 7316876747366973440. Current Snapshot: MvccSnapshot[applied={T|T < 7316876747573919744}]
I20260810 07:27:42.223598 1760 mvcc.cc:205] Tried to move back new op lower bound from 7316876747573919744 to 7316876747366973440. Current Snapshot: MvccSnapshot[applied={T|T < 7316876747573919744}]
I20260810 07:27:42.230322 1758 mvcc.cc:205] Tried to move back new op lower bound from 7316876747573919744 to 7316876747366973440. Current Snapshot: MvccSnapshot[applied={T|T < 7316876747573919744}]
I20260810 07:27:42.232710 977 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:27:42.232962 977 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:27:42.235926 977 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36bc2b56025342ec869b7536820e2c80: Sending DeleteTablet for 3 replicas of tablet dadfc625268741828fef54fdc41e9221
I20260810 07:27:42.237375 1568 tablet_service.cc:1514] Processing DeleteTablet for tablet dadfc625268741828fef54fdc41e9221 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:27:42 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:57754
I20260810 07:27:42.237875 1172 tablet_service.cc:1514] Processing DeleteTablet for tablet dadfc625268741828fef54fdc41e9221 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:27:42 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:50642
I20260810 07:27:42.238394 1368 tablet_service.cc:1514] Processing DeleteTablet for tablet dadfc625268741828fef54fdc41e9221 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:27:42 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:53254
I20260810 07:27:42.239882 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 1035
I20260810 07:27:42.250365 1770 tablet_replica.cc:330] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4: stopping tablet replica
I20260810 07:27:42.257664 1773 tablet_replica.cc:330] T dadfc625268741828fef54fdc41e9221 P b50dcac920074471a959d6ee8e5eb74b: stopping tablet replica
I20260810 07:27:42.258270 1770 raft_consensus.cc:2229] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:27:42.258710 1773 raft_consensus.cc:2229] T dadfc625268741828fef54fdc41e9221 P b50dcac920074471a959d6ee8e5eb74b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:27:42.259135 1773 raft_consensus.cc:2258] T dadfc625268741828fef54fdc41e9221 P b50dcac920074471a959d6ee8e5eb74b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:27:42.259613 1770 raft_consensus.cc:2258] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:27:42.261726 1770 ts_tablet_manager.cc:1905] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:27:42.263700 1773 ts_tablet_manager.cc:1905] T dadfc625268741828fef54fdc41e9221 P b50dcac920074471a959d6ee8e5eb74b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:27:42.265571 1770 ts_tablet_manager.cc:1918] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:27:42.266640 1773 ts_tablet_manager.cc:1918] T dadfc625268741828fef54fdc41e9221 P b50dcac920074471a959d6ee8e5eb74b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:27:42.266803 1773 log.cc:1196] T dadfc625268741828fef54fdc41e9221 P b50dcac920074471a959d6ee8e5eb74b: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-1/wal/wals/dadfc625268741828fef54fdc41e9221
I20260810 07:27:42.267223 1773 ts_tablet_manager.cc:1939] T dadfc625268741828fef54fdc41e9221 P b50dcac920074471a959d6ee8e5eb74b: Deleting consensus metadata
I20260810 07:27:42.268724 958 catalog_manager.cc:4841] TS b50dcac920074471a959d6ee8e5eb74b (127.12.157.194:46147): tablet dadfc625268741828fef54fdc41e9221 (table test-table [id=7ffe334c13124018916c6bf6f42f04f8]) successfully deleted
I20260810 07:27:42.269049 1770 log.cc:1196] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786346694975122-12919-0/minicluster-data/ts-2/wal/wals/dadfc625268741828fef54fdc41e9221
I20260810 07:27:42.269505 1770 ts_tablet_manager.cc:1939] T dadfc625268741828fef54fdc41e9221 P d7f6d311b26e4b20bb5df7a81fb2a7f4: Deleting consensus metadata
W20260810 07:27:42.269713 958 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.157.193:42479 (error 108)
I20260810 07:27:42.270745 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 1270
I20260810 07:27:42.270762 957 catalog_manager.cc:4841] TS d7f6d311b26e4b20bb5df7a81fb2a7f4 (127.12.157.195:34459): tablet dadfc625268741828fef54fdc41e9221 (table test-table [id=7ffe334c13124018916c6bf6f42f04f8]) successfully deleted
I20260810 07:27:42.291522 1733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:42.290384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:42479 (local address 127.0.0.1:50652)
0810 07:27:42.291292 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:42.291394 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.193:42479: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":787}
W20260810 07:27:42.291833 958 catalog_manager.cc:4584] TS b7b301973de3499da5d5284c9bae437c (127.12.157.193:42479): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dadfc625268741828fef54fdc41e9221: Network error: Client connection negotiation failed: client connection to 127.12.157.193:42479: connect: Connection refused (error 111)
I20260810 07:27:42.300503 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 1471
I20260810 07:27:42.317271 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 934
I20260810 07:27:42.344493 12919 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 07:27:42.690109 12919 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 07:27:42.690277 12919 mini_ranger.cc:66] Stopping Ranger...
I20260810 07:27:43.046736 12919 mini_ranger.cc:68] Stopped Ranger
2026-08-10 07:27:43.047 UTC [32008] LOG: received smart shutdown request
2026-08-10 07:27:43.051 UTC [32008] LOG: background worker "logical replication launcher" (PID 32019) exited with exit code 1
2026-08-10 07:27:43.051 UTC [32012] LOG: shutting down
2026-08-10 07:27:43.214 UTC [32008] LOG: database system is shut down
2026-08-10T07:27:43Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50095 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:27:43 dist-test-slave-ffrd krb5kdc[1791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:27:43 dist-test-slave-ffrd krb5kdc[1791](info): set up 2 sockets
Aug 10 07:27:43 dist-test-slave-ffrd krb5kdc[1791](info): commencing operation
krb5kdc: starting...
W20260810 07:27:44.633296 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.269s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:27:44 dist-test-slave-ffrd krb5kdc[1791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:27:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:27:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:27:44.765579 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:41661
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:35971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:41661,127.12.157.253:40585,127.12.157.252:36451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:44.882961 1811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:44.883323 1811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:44.883406 1811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:44.887496 1811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:27:44.887609 1811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:44.887650 1811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:44.887691 1811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:27:44.887730 1811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:27:44.893261 1811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:35971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:41661,127.12.157.253:40585,127.12.157.252:36451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:41661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:27:44.894443 1811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:44.895521 1811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:44.904155 1825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:44.904155 1822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:44.904157 1821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:44.904419 1811 server_base.cc:1034] running on GCE node
I20260810 07:27:44.905232 1811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:44.906407 1811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:44.907603 1811 hybrid_clock.cc:648] HybridClock initialized: now 1786346864907584 us; error 32 us; skew 500 ppm
Aug 10 07:27:44 dist-test-slave-ffrd krb5kdc[1791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346864, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:44.911152 1811 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:27:44.912195 1811 webserver.cc:460] Webserver started at http://127.12.157.254:33459/ using document root <none> and password file <none>
I20260810 07:27:44.912814 1811 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:44.912916 1811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:44.913193 1811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:27:44.915349 1811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "776bcd586cd24e9cac41005fb1037c13"
format_stamp: "Formatted at 2026-08-10 07:27:44 on dist-test-slave-ffrd"
server_key: "a83826c15ac5acc6b405afc5412b0c13"
server_key_iv: "3f2ead4b16b712bee94e402d7b98b48e"
server_key_version: "encryptionkey@0"
I20260810 07:27:44.915954 1811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "776bcd586cd24e9cac41005fb1037c13"
format_stamp: "Formatted at 2026-08-10 07:27:44 on dist-test-slave-ffrd"
server_key: "a83826c15ac5acc6b405afc5412b0c13"
server_key_iv: "3f2ead4b16b712bee94e402d7b98b48e"
server_key_version: "encryptionkey@0"
I20260810 07:27:44.919759 1811 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 07:27:44.922271 1837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:44.923521 1811 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 07:27:44.923652 1811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "776bcd586cd24e9cac41005fb1037c13"
format_stamp: "Formatted at 2026-08-10 07:27:44 on dist-test-slave-ffrd"
server_key: "a83826c15ac5acc6b405afc5412b0c13"
server_key_iv: "3f2ead4b16b712bee94e402d7b98b48e"
server_key_version: "encryptionkey@0"
I20260810 07:27:44.923763 1811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:44.944857 1811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:44.948527 1811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:44.948752 1811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:44.956362 1811 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:41661
I20260810 07:27:44.956773 1908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:41661 every 8 connection(s)
I20260810 07:27:44.957576 1811 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:27:44.960646 1909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:44.963338 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 1811
I20260810 07:27:44.963455 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:27:44.963721 12919 external_mini_cluster.cc:1428] Setting key 82120ceb70ef86ec9e2f85ef6b012639
I20260810 07:27:44.965178 1909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41661 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.157.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.253@KRBTEST.COM" created.
Aug 10 07:27:44 dist-test-slave-ffrd krb5kdc[1791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346864, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:27:44.981886 1915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:44.968009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:41661 (local address 127.0.0.1:46492)
0810 07:27:44.968541 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:44.968586 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:44.969563 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:44.970275 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:44.970281 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:44.970736 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:44.970975 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:44.970988 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:44.972160 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:44.972162 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:44.972901 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:44.972910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:44.973685 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:44.973688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:44.973820 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:44.974400 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:44.974416 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:44.976746 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:44.978868 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:44.978876 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:44.978887 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:44.979113 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:44.979439 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:44.979443 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:44.979445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:44.979561 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:44.980162 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:44.980168 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:44.980318 (+ 150us) client_negotiation.cc:769] Sending connection context
0810 07:27:44.980487 (+ 169us) client_negotiation.cc:240] Negotiation successful
0810 07:27:44.980864 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":109,"thread_start_us":166,"threads_started":1}
I20260810 07:27:44.981886 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:44.969006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46492 (local address 127.12.157.254:41661)
0810 07:27:44.969310 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:44.969322 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:44.969372 (+ 50us) server_negotiation.cc:407] Connection header received
0810 07:27:44.969896 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:44.969901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:44.970076 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:44.970265 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:44.971125 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:44.972012 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:44.973094 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:44.973532 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:44.973700 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:44.976983 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:44.976992 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:44.977009 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:44.977031 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:44.978718 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:44.979226 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:44.979232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:44.979234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:44.979343 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:44.979654 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:44.979659 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:44.979660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:44.980039 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:44.980180 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:44.980562 (+ 382us) server_negotiation.cc:299] Negotiation successful
0810 07:27:44.980817 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":201,"thread_start_us":97,"threads_started":1}
I20260810 07:27:44.985121 1909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.253" port: 40585 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:27:44.985924 1915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:44.985624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:40585 (local address 127.0.0.1:36466)
0810 07:27:44.985767 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:44.985852 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.253:40585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260810 07:27:44.986125 1839 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.157.253:40585: connect: Connection refused (error 111)
I20260810 07:27:44.987046 1915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:44.986825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:40585 (local address 127.0.0.1:36474)
0810 07:27:44.986940 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:44.986994 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.253:40585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 07:27:44.987190 1909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.157.253:40585: Network error: Client connection negotiation failed: client connection to 127.12.157.253:40585: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.253.keytab.
Entry for principal kudu/127.12.157.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.253.keytab.
I20260810 07:27:44.992779 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.253:40585
--webserver_interface=127.12.157.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.253.keytab
--principal=kudu/127.12.157.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:35971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:41661,127.12.157.253:40585,127.12.157.252:36451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:27:45.036482 1909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.157.253" port: 40585 } attempt: 1
I20260810 07:27:45.037518 1915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.037171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:40585 (local address 127.0.0.1:36482)
0810 07:27:45.037353 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:45.037427 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.253:40585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 07:27:45.038342 1915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.038107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:40585 (local address 127.0.0.1:36486)
0810 07:27:45.038222 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:45.038287 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.253:40585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 07:27:45.038496 1909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.157.253:40585: Network error: Client connection negotiation failed: client connection to 127.12.157.253:40585: connect: Connection refused (error 111)
I20260810 07:27:45.106693 1909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.157.253" port: 40585 } attempt: 2
I20260810 07:27:45.107609 1915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.107289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:40585 (local address 127.0.0.1:36498)
0810 07:27:45.107463 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:45.107534 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.253:40585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260810 07:27:45.108373 1915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.108157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:40585 (local address 127.0.0.1:36502)
0810 07:27:45.108265 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:45.108314 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.253:40585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 07:27:45.108597 1909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.157.253:40585: Network error: Client connection negotiation failed: client connection to 127.12.157.253:40585: connect: Connection refused (error 111)
W20260810 07:27:45.114163 1922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:45.114393 1922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:45.114467 1922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:45.118465 1922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:27:45.118516 1922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:45.118535 1922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:45.118552 1922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:27:45.118599 1922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:27:45.123843 1922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:35971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:41661,127.12.157.253:40585,127.12.157.252:36451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.253.keytab
--principal=kudu/127.12.157.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.253:40585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.157.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:27:45.124986 1922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:45.126030 1922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:45.134754 1931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:45.134810 1938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:45.134840 1930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:45.135103 1922 server_base.cc:1034] running on GCE node
I20260810 07:27:45.135790 1922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:45.137068 1922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:45.138298 1922 hybrid_clock.cc:648] HybridClock initialized: now 1786346865138274 us; error 44 us; skew 500 ppm
Aug 10 07:27:45 dist-test-slave-ffrd krb5kdc[1791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346865, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:45.141600 1922 init.cc:377] Logged in from keytab as kudu/127.12.157.253@KRBTEST.COM (short username kudu)
I20260810 07:27:45.142506 1922 webserver.cc:460] Webserver started at http://127.12.157.253:43493/ using document root <none> and password file <none>
I20260810 07:27:45.143051 1922 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:45.143110 1922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:45.143283 1922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:27:45.145470 1922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-1/data/instance:
uuid: "7a029fb8f8034c5e8d1eb568f3773d2e"
format_stamp: "Formatted at 2026-08-10 07:27:45 on dist-test-slave-ffrd"
server_key: "2c900ce9f18f49dde06a88150a476da9"
server_key_iv: "046154e52a97e3b9383dce4cfc496ab3"
server_key_version: "encryptionkey@0"
I20260810 07:27:45.145994 1922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-1/wal/instance:
uuid: "7a029fb8f8034c5e8d1eb568f3773d2e"
format_stamp: "Formatted at 2026-08-10 07:27:45 on dist-test-slave-ffrd"
server_key: "2c900ce9f18f49dde06a88150a476da9"
server_key_iv: "046154e52a97e3b9383dce4cfc496ab3"
server_key_version: "encryptionkey@0"
I20260810 07:27:45.149833 1922 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 07:27:45.152267 1947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:45.153693 1922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:27:45.153883 1922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-1/wal
uuid: "7a029fb8f8034c5e8d1eb568f3773d2e"
format_stamp: "Formatted at 2026-08-10 07:27:45 on dist-test-slave-ffrd"
server_key: "2c900ce9f18f49dde06a88150a476da9"
server_key_iv: "046154e52a97e3b9383dce4cfc496ab3"
server_key_version: "encryptionkey@0"
I20260810 07:27:45.153987 1922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:45.168208 1922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:45.171660 1922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:45.171886 1922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:45.179147 1922 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.253:40585
I20260810 07:27:45.179487 2028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.253:40585 every 8 connection(s)
I20260810 07:27:45.180204 1922 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-1/data/info.pb
I20260810 07:27:45.181255 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 1922
I20260810 07:27:45.181463 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-1/wal/instance
I20260810 07:27:45.181811 12919 external_mini_cluster.cc:1428] Setting key 06ba26c3dba563f7ca40a23f206d4783
I20260810 07:27:45.184010 2029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:45.188086 2029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41661 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.157.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.252@KRBTEST.COM" created.
Aug 10 07:27:45 dist-test-slave-ffrd krb5kdc[1791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346865, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.253@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:27:45.199782 1909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.157.253" port: 40585 } attempt: 3
I20260810 07:27:45.204262 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.190294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46506 (local address 127.12.157.254:41661)
0810 07:27:45.190474 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:45.190480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:45.191314 (+ 834us) server_negotiation.cc:407] Connection header received
0810 07:27:45.192405 (+ 1091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:45.192409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:45.192473 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:45.192556 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:45.194104 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.194654 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.195799 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.196056 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.196174 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:45.199113 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:45.199117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:45.199119 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:45.199145 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:45.201858 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:45.202533 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:45.202537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:45.202539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:45.202634 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:45.203042 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:45.203046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:45.203047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:45.203266 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:45.203368 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:45.203964 (+ 596us) server_negotiation.cc:299] Negotiation successful
0810 07:27:45.204090 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 07:27:45.205106 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.190629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:41661 (local address 127.0.0.1:46506)
0810 07:27:45.191158 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:45.191202 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:45.192174 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:45.192748 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:45.192755 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:45.193190 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:45.193944 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:45.193962 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.194792 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.194796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:45.195661 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:45.195670 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.196222 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.196226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:45.196397 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:45.197100 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:45.197126 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:45.198915 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:45.202010 (+ 3095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:45.202017 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:45.202032 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:45.202414 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:45.202746 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:45.202749 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:45.202751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:45.202941 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:45.203396 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:45.203403 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:45.203658 (+ 255us) client_negotiation.cc:769] Sending connection context
0810 07:27:45.203832 (+ 174us) client_negotiation.cc:240] Negotiation successful
0810 07:27:45.204132 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":120,"threads_started":1}
Aug 10 07:27:45 dist-test-slave-ffrd krb5kdc[1791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346864, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for kudu/127.12.157.253@KRBTEST.COM
I20260810 07:27:45.206682 2029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.253" port: 40585 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:27:45.210886 1915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.200347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:40585 (local address 127.0.0.1:36516)
0810 07:27:45.200570 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:45.200584 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:45.200671 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:45.201390 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:45.201393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:45.201687 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:45.201793 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:45.201799 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.202909 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.202915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:45.203637 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:45.203646 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.204594 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.204597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:45.204699 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:45.205162 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:45.205201 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:45.206788 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:45.209169 (+ 2381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:45.209172 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:45.209175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:45.209440 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:45.209723 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:45.209726 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:45.209728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:45.209831 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:45.210326 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:45.210328 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:45.210459 (+ 131us) client_negotiation.cc:769] Sending connection context
0810 07:27:45.210565 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 07:27:45.210678 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260810 07:27:45.211115 2034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.200463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.12.157.253:40585)
0810 07:27:45.200936 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:45.200997 (+ 61us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:45.201027 (+ 30us) server_negotiation.cc:407] Connection header received
0810 07:27:45.201080 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:45.201085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:45.201258 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:45.201407 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:45.201914 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.202799 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.203767 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.204390 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.205004 (+ 614us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:45.206946 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:45.206956 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:45.206968 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:45.206991 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:45.209065 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:45.209540 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:45.209545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:45.209546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:45.209626 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:45.209920 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:45.209923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:45.209925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:45.210212 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:45.210349 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:45.210695 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 07:27:45.210886 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.252.keytab.
Entry for principal kudu/127.12.157.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.252.keytab.
I20260810 07:27:45.212565 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.252:36451
--webserver_interface=127.12.157.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.252.keytab
--principal=kudu/127.12.157.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:35971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:41661,127.12.157.253:40585,127.12.157.252:36451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 07:27:45 dist-test-slave-ffrd krb5kdc[1791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346865, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.253@KRBTEST.COM for kudu/127.12.157.253@KRBTEST.COM
I20260810 07:27:45.214638 1909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.252" port: 36451 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:27:45.215427 1915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.215172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:36451 (local address 127.0.0.1:36438)
0810 07:27:45.215307 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:45.215364 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 07:27:45.216347 1915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.216057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:36451 (local address 127.0.0.1:36448)
0810 07:27:45.216239 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:45.216291 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260810 07:27:45.216512 1909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.157.252:36451: Network error: Client connection negotiation failed: client connection to 127.12.157.252:36451: connect: Connection refused (error 111)
I20260810 07:27:45.218098 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.207305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:40585 (local address 127.0.0.1:36518)
0810 07:27:45.207455 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:45.207471 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:45.207547 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:45.208408 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:45.208413 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:45.208570 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:45.208681 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:45.208689 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.209909 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.209912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:45.210667 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:45.210676 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.211329 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.211335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:45.211459 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:45.211954 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:45.211966 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:45.213533 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:45.216498 (+ 2965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:45.216504 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:45.216507 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:45.216746 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:45.217100 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:45.217103 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:45.217104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:45.217188 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:45.217580 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:45.217583 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:45.217674 (+ 91us) client_negotiation.cc:769] Sending connection context
0810 07:27:45.217777 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 07:27:45.217932 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 07:27:45.218142 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.207639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36518 (local address 127.12.157.253:40585)
0810 07:27:45.208038 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:45.208050 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:45.208065 (+ 15us) server_negotiation.cc:407] Connection header received
0810 07:27:45.208100 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:45.208106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:45.208154 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:45.208261 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:45.208824 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.209783 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.210850 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.211203 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.211689 (+ 486us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:45.214147 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:45.214151 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:45.214152 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:45.214173 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:45.216067 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:45.216845 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:45.216850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:45.216854 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:45.216988 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:45.217284 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:45.217287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:45.217289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:45.217475 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:45.217580 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:45.217834 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 07:27:45.217993 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":318,"thread_start_us":192,"threads_started":1}
I20260810 07:27:45.218744 2029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.252" port: 36451 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:27:45.219503 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.219166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:36451 (local address 127.0.0.1:36454)
0810 07:27:45.219314 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:45.219415 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 07:27:45.219749 1950 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.157.252:36451: connect: Connection refused (error 111)
I20260810 07:27:45.220709 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.220426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:36451 (local address 127.0.0.1:36460)
0810 07:27:45.220567 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:45.220631 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 07:27:45.220917 2029 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.157.252:36451: Network error: Client connection negotiation failed: client connection to 127.12.157.252:36451: connect: Connection refused (error 111)
I20260810 07:27:45.247695 1909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.157.252" port: 36451 } attempt: 1
I20260810 07:27:45.248631 1915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.248268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:36451 (local address 127.0.0.1:36472)
0810 07:27:45.248459 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:45.248535 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 07:27:45.249469 1915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.249203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:36451 (local address 127.0.0.1:36474)
0810 07:27:45.249318 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:45.249382 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 07:27:45.249657 1909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.157.252:36451: Network error: Client connection negotiation failed: client connection to 127.12.157.252:36451: connect: Connection refused (error 111)
I20260810 07:27:45.270252 2029 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.157.252" port: 36451 } attempt: 1
I20260810 07:27:45.271173 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.270831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:36451 (local address 127.0.0.1:36488)
0810 07:27:45.271008 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:45.271080 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 07:27:45.271929 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.271704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:36451 (local address 127.0.0.1:36496)
0810 07:27:45.271817 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:45.271869 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 07:27:45.272080 2029 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.157.252:36451: Network error: Client connection negotiation failed: client connection to 127.12.157.252:36451: connect: Connection refused (error 111)
I20260810 07:27:45.324883 1909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.157.252" port: 36451 } attempt: 2
I20260810 07:27:45.325911 1915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.325543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:36451 (local address 127.0.0.1:36500)
0810 07:27:45.325732 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:45.325812 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 07:27:45.327019 1915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.326755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:36451 (local address 127.0.0.1:36502)
0810 07:27:45.326885 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:45.326942 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 07:27:45.327175 1909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.157.252:36451: Network error: Client connection negotiation failed: client connection to 127.12.157.252:36451: connect: Connection refused (error 111)
W20260810 07:27:45.335791 2038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:45.336084 2038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:45.336167 2038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:45.340315 2038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
I20260810 07:27:45.340265 2029 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.157.252" port: 36451 } attempt: 2
W20260810 07:27:45.340408 2038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:45.340448 2038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:45.340508 2038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:27:45.340566 2038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:27:45.341181 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.340815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:36451 (local address 127.0.0.1:36510)
0810 07:27:45.341012 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:45.341085 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 07:27:45.342051 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.341797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:36451 (local address 127.0.0.1:36516)
0810 07:27:45.341925 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:45.341980 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:36451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 07:27:45.342198 2029 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.157.252:36451: Network error: Client connection negotiation failed: client connection to 127.12.157.252:36451: connect: Connection refused (error 111)
I20260810 07:27:45.346560 2038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:35971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:41661,127.12.157.253:40585,127.12.157.252:36451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.252.keytab
--principal=kudu/127.12.157.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.252:36451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.157.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:27:45.347645 2038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:45.348670 2038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:45.357852 2051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:45.357969 2047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:45.357892 2046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:45.358341 2038 server_base.cc:1034] running on GCE node
I20260810 07:27:45.359028 2038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:45.360160 2038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:45.361388 2038 hybrid_clock.cc:648] HybridClock initialized: now 1786346865361364 us; error 37 us; skew 500 ppm
Aug 10 07:27:45 dist-test-slave-ffrd krb5kdc[1791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346865, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:45.364315 2038 init.cc:377] Logged in from keytab as kudu/127.12.157.252@KRBTEST.COM (short username kudu)
I20260810 07:27:45.365375 2038 webserver.cc:460] Webserver started at http://127.12.157.252:41295/ using document root <none> and password file <none>
I20260810 07:27:45.365944 2038 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:45.366039 2038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:45.366286 2038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:27:45.368376 2038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-2/data/instance:
uuid: "234b604f10bd489387f5583435414504"
format_stamp: "Formatted at 2026-08-10 07:27:45 on dist-test-slave-ffrd"
server_key: "d38a28a275b8d0aaa573aefecf84ced9"
server_key_iv: "fc2221423ea6f1cba1a19175e59741a9"
server_key_version: "encryptionkey@0"
I20260810 07:27:45.369029 2038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-2/wal/instance:
uuid: "234b604f10bd489387f5583435414504"
format_stamp: "Formatted at 2026-08-10 07:27:45 on dist-test-slave-ffrd"
server_key: "d38a28a275b8d0aaa573aefecf84ced9"
server_key_iv: "fc2221423ea6f1cba1a19175e59741a9"
server_key_version: "encryptionkey@0"
I20260810 07:27:45.373054 2038 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 07:27:45.375708 2061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:45.377156 2038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:27:45.377303 2038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-2/wal
uuid: "234b604f10bd489387f5583435414504"
format_stamp: "Formatted at 2026-08-10 07:27:45 on dist-test-slave-ffrd"
server_key: "d38a28a275b8d0aaa573aefecf84ced9"
server_key_iv: "fc2221423ea6f1cba1a19175e59741a9"
server_key_version: "encryptionkey@0"
I20260810 07:27:45.377436 2038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:45.400727 2038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:45.404582 2038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:45.404827 2038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:45.412745 2038 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.252:36451
I20260810 07:27:45.413065 2144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.252:36451 every 8 connection(s)
I20260810 07:27:45.413862 2038 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-2/data/info.pb
I20260810 07:27:45.416929 2145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:45.420830 2145 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41661 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:27:45.421588 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 2038
I20260810 07:27:45.421682 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786346694975122-12919-0/minicluster-data/master-2/wal/instance
I20260810 07:27:45.421952 12919 external_mini_cluster.cc:1428] Setting key f9a002885f92fa808f5984d4e5aee4f3
I20260810 07:27:45.426380 1909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.157.252" port: 36451 } attempt: 3
Aug 10 07:27:45 dist-test-slave-ffrd krb5kdc[1791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346865, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.252@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
Aug 10 07:27:45 dist-test-slave-ffrd krb5kdc[1791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.252@KRBTEST.COM
Aug 10 07:27:45 dist-test-slave-ffrd krb5kdc[1791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346864, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for kudu/127.12.157.252@KRBTEST.COM
I20260810 07:27:45.433363 2029 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.157.252" port: 36451 } attempt: 3
I20260810 07:27:45.437668 1915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.426839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:36451 (local address 127.0.0.1:36536)
0810 07:27:45.427004 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:45.427018 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:45.427087 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:45.427401 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:45.427406 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:45.427588 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:45.427677 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:45.427683 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.428733 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.428736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:45.429582 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:45.429595 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.430217 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.430224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:45.430324 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:45.430823 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:45.430837 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:45.432121 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:45.435868 (+ 3747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:45.435871 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:45.435873 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:45.436121 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:45.436444 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:45.436447 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:45.436449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:45.436541 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:45.437130 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:45.437134 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:45.437248 (+ 114us) client_negotiation.cc:769] Sending connection context
0810 07:27:45.437336 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 07:27:45.437474 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"spinlock_wait_cycles":8960}
I20260810 07:27:45.438107 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.423088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46512 (local address 127.12.157.254:41661)
0810 07:27:45.423207 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:45.423212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:45.423929 (+ 717us) server_negotiation.cc:407] Connection header received
0810 07:27:45.424757 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:45.424761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:45.424864 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:45.424996 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:45.426037 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.426562 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.428081 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.428284 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.428432 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:45.432987 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:45.432991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:45.432993 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:45.433019 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:45.434690 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:45.435582 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:45.435585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:45.435587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:45.435680 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:45.436105 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:45.436109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:45.436111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:45.436465 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:45.436542 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:45.437254 (+ 712us) server_negotiation.cc:299] Negotiation successful
0810 07:27:45.437400 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 07:27:45.439208 2154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.423287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:41661 (local address 127.0.0.1:46512)
0810 07:27:45.423793 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:45.423833 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:45.424541 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:45.425057 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:45.425063 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:45.425499 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:45.425881 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:45.425900 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.426700 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.426707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:45.427927 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:45.427936 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.428474 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.428478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:45.428639 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:45.429431 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:45.429457 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:45.432785 (+ 3328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:45.434992 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:45.435001 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:45.435014 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:45.435455 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:45.435802 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:45.435805 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:45.435807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:45.435988 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:45.436622 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:45.436628 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:45.436999 (+ 371us) client_negotiation.cc:769] Sending connection context
0810 07:27:45.437537 (+ 538us) client_negotiation.cc:240] Negotiation successful
0810 07:27:45.437895 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":50,"spinlock_wait_cycles":7040,"thread_start_us":171,"threads_started":1}
I20260810 07:27:45.439208 2153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.423287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36532 (local address 127.12.157.252:36451)
0810 07:27:45.423693 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:45.423737 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:45.423779 (+ 42us) server_negotiation.cc:407] Connection header received
0810 07:27:45.424524 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:45.424543 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:45.424746 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:45.424888 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:45.425754 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.427005 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.428109 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.428620 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.428741 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:45.431949 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:45.431957 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:45.431972 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:45.432003 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:45.435586 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:45.436122 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:45.436127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:45.436129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:45.436199 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:45.436525 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:45.436527 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:45.436529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:45.437003 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:45.437907 (+ 904us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:45.438007 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 07:27:45.438161 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":274,"thread_start_us":116,"threads_started":1}
Aug 10 07:27:45 dist-test-slave-ffrd krb5kdc[1791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346865, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.253@KRBTEST.COM for kudu/127.12.157.252@KRBTEST.COM
I20260810 07:27:45.439234 2155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.426874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36536 (local address 127.12.157.252:36451)
0810 07:27:45.427167 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:45.427175 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:45.427187 (+ 12us) server_negotiation.cc:407] Connection header received
0810 07:27:45.427219 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:45.427222 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:45.427277 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:45.427365 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:45.427852 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.428620 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.429720 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.430067 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.430206 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:45.432285 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:45.432289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:45.432291 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:45.432322 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:45.435745 (+ 3423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:45.436234 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:45.436239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:45.436241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:45.436343 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:45.436650 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:45.436653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:45.436655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:45.437006 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:45.438798 (+ 1792us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:45.438886 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 07:27:45.439031 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":218,"thread_start_us":109,"threads_started":1}
I20260810 07:27:45.441046 2145 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.253" port: 40585 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:27:45.445113 1909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 776bcd586cd24e9cac41005fb1037c13: Bootstrap starting.
I20260810 07:27:45.446666 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.434346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:36451 (local address 127.0.0.1:36550)
0810 07:27:45.434598 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:45.434610 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:45.434679 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:45.434938 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:45.434941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:45.435123 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:45.435217 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:45.435223 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.436311 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.436314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:45.437164 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:45.437175 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.437826 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.437833 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:45.437913 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:45.438369 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:45.438388 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:45.439904 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:45.444990 (+ 5086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:45.444994 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:45.444996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:45.445258 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:45.445604 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:45.445607 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:45.445609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:45.445701 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:45.446142 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:45.446145 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:45.446243 (+ 98us) client_negotiation.cc:769] Sending connection context
0810 07:27:45.446325 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 07:27:45.446485 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260810 07:27:45.447276 2157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.434345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36550 (local address 127.12.157.252:36451)
0810 07:27:45.434705 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:45.434714 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:45.434730 (+ 16us) server_negotiation.cc:407] Connection header received
0810 07:27:45.434769 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:45.434775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:45.434825 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:45.434906 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:45.435347 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.436179 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.437308 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.437668 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.438135 (+ 467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:45.440062 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:45.440066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:45.440069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:45.440093 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:45.444825 (+ 4732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:45.445390 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:45.445393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:45.445395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:45.445491 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:45.445816 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:45.445819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:45.445821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:45.446023 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:45.446932 (+ 909us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:45.447009 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 07:27:45.447122 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":115,"threads_started":1}
I20260810 07:27:45.448177 1909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 776bcd586cd24e9cac41005fb1037c13: Neither blocks nor log segments found. Creating new log.
Aug 10 07:27:45 dist-test-slave-ffrd krb5kdc[1791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
Aug 10 07:27:45 dist-test-slave-ffrd krb5kdc[1791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346865, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.252@KRBTEST.COM for kudu/127.12.157.253@KRBTEST.COM
I20260810 07:27:45.448889 1909 log.cc:824] T 00000000000000000000000000000000 P 776bcd586cd24e9cac41005fb1037c13: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:45.451529 2029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e: Bootstrap starting.
I20260810 07:27:45.452772 2154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.441745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:40585 (local address 127.0.0.1:36526)
0810 07:27:45.442085 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:45.442101 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:45.442198 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:45.443480 (+ 1282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:45.443483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:45.443680 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:45.443792 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:45.443799 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.445087 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.445090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:45.445873 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:45.445887 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.446967 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.446971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:45.447108 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:45.447665 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:45.447683 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:45.449047 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:45.451021 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:45.451023 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:45.451025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:45.451220 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:45.451602 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:45.451605 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:45.451607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:45.451694 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:45.452260 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:45.452263 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:45.452360 (+ 97us) client_negotiation.cc:769] Sending connection context
0810 07:27:45.452474 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 07:27:45.452613 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260810 07:27:45.452857 1909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 776bcd586cd24e9cac41005fb1037c13: No bootstrap required, opened a new log
I20260810 07:27:45.453501 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.443144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36526 (local address 127.12.157.253:40585)
0810 07:27:45.443265 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:45.443269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:45.443282 (+ 13us) server_negotiation.cc:407] Connection header received
0810 07:27:45.443316 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:45.443319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:45.443355 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:45.443441 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:45.444185 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.444929 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.446015 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.446712 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.446797 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:45.449204 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:45.449208 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:45.449210 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:45.449237 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:45.450859 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:45.451382 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:45.451400 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:45.451405 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:45.451493 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:45.451851 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:45.451854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:45.451856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:45.452115 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:45.452213 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:45.453230 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0810 07:27:45.453353 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 07:27:45.453725 2029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:45.454128 2145 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.252" port: 36451 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:27:45.454540 2029 log.cc:824] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:45.457592 2029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e: No bootstrap required, opened a new log
I20260810 07:27:45.457479 1909 raft_consensus.cc:348] T 00000000000000000000000000000000 P 776bcd586cd24e9cac41005fb1037c13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "776bcd586cd24e9cac41005fb1037c13" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41661 } } peers { permanent_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 40585 } } peers { permanent_uuid: "234b604f10bd489387f5583435414504" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 36451 } }
I20260810 07:27:45.457743 1909 raft_consensus.cc:374] T 00000000000000000000000000000000 P 776bcd586cd24e9cac41005fb1037c13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:45.457823 1909 raft_consensus.cc:729] T 00000000000000000000000000000000 P 776bcd586cd24e9cac41005fb1037c13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 776bcd586cd24e9cac41005fb1037c13, State: Initialized, Role: FOLLOWER
I20260810 07:27:45.458367 1909 consensus_queue.cc:260] T 00000000000000000000000000000000 P 776bcd586cd24e9cac41005fb1037c13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "776bcd586cd24e9cac41005fb1037c13" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41661 } } peers { permanent_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 40585 } } peers { permanent_uuid: "234b604f10bd489387f5583435414504" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 36451 } }
I20260810 07:27:45.459393 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.442513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46516 (local address 127.12.157.254:41661)
0810 07:27:45.442684 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:45.442689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:45.442703 (+ 14us) server_negotiation.cc:407] Connection header received
0810 07:27:45.442789 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:45.442792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:45.442848 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:45.442923 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:45.443669 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.444432 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.445515 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.445848 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.445949 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:45.455167 (+ 9218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:45.455172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:45.455174 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:45.455202 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:45.457026 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:45.457563 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:45.457567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:45.457568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:45.457660 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:45.457977 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:45.457980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:45.457982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:45.458207 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:45.458382 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:45.459106 (+ 724us) server_negotiation.cc:299] Negotiation successful
0810 07:27:45.459232 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 07:27:45.459857 1909 sys_catalog.cc:564] T 00000000000000000000000000000000 P 776bcd586cd24e9cac41005fb1037c13 [sys.catalog]: configured and running, proceeding with master startup.
Aug 10 07:27:45 dist-test-slave-ffrd krb5kdc[1791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346865, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.252@KRBTEST.COM for kudu/127.12.157.252@KRBTEST.COM
I20260810 07:27:45.462561 2029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "776bcd586cd24e9cac41005fb1037c13" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41661 } } peers { permanent_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 40585 } } peers { permanent_uuid: "234b604f10bd489387f5583435414504" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 36451 } }
I20260810 07:27:45.462847 2029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:45.462939 2029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a029fb8f8034c5e8d1eb568f3773d2e, State: Initialized, Role: FOLLOWER
I20260810 07:27:45.463439 2029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "776bcd586cd24e9cac41005fb1037c13" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41661 } } peers { permanent_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 40585 } } peers { permanent_uuid: "234b604f10bd489387f5583435414504" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 36451 } }
I20260810 07:27:45.464784 2029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:27:45.465605 2154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.454668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:36451 (local address 127.0.0.1:36556)
0810 07:27:45.454879 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:45.454891 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:45.454961 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:45.455269 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:45.455273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:45.455752 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:45.455859 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:45.455866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.457309 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.457313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:45.458073 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:45.458084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.458782 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.458785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:45.458860 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:45.459310 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:45.459325 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:45.461187 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:45.463640 (+ 2453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:45.463643 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:45.463645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:45.463904 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:45.464213 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:45.464216 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:45.464218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:45.464304 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:45.464772 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:45.464778 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:45.464868 (+ 90us) client_negotiation.cc:769] Sending connection context
0810 07:27:45.464994 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 07:27:45.465447 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260810 07:27:45.466192 2157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.454787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36556 (local address 127.12.157.252:36451)
0810 07:27:45.455088 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:45.455092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:45.455102 (+ 10us) server_negotiation.cc:407] Connection header received
0810 07:27:45.455132 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:45.455137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:45.455174 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:45.455962 (+ 788us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:45.456035 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.457170 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.458259 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.458661 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.461620 (+ 2959us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:45.461746 (+ 126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:45.461753 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:45.461759 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:45.461785 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:45.463530 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:45.464030 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:45.464035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:45.464039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:45.464120 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:45.464424 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:45.464428 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:45.464430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:45.464663 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:45.465885 (+ 1222us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:45.465960 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 07:27:45.466073 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
I20260810 07:27:45.466281 2161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 776bcd586cd24e9cac41005fb1037c13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "776bcd586cd24e9cac41005fb1037c13" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41661 } } peers { permanent_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 40585 } } peers { permanent_uuid: "234b604f10bd489387f5583435414504" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 36451 } } }
I20260810 07:27:45.466524 2161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 776bcd586cd24e9cac41005fb1037c13 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:27:45.467072 2180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "776bcd586cd24e9cac41005fb1037c13" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41661 } } peers { permanent_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 40585 } } peers { permanent_uuid: "234b604f10bd489387f5583435414504" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 36451 } } }
I20260810 07:27:45.469313 2145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 234b604f10bd489387f5583435414504: Bootstrap starting.
W20260810 07:27:45.469733 2178 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 776bcd586cd24e9cac41005fb1037c13: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:27:45.469797 2178 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:27:45.470141 2180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:27:45.466606 1811 master_runner.cc:186] Error getting master registration for 127.12.157.253:40585: OK, instance_id { permanent_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" instance_seqno: 1786346865177073 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 07:27:45.470942 1811 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 10 07:27:45 dist-test-slave-ffrd krb5kdc[1791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.253@KRBTEST.COM
I20260810 07:27:45.472030 2145 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 234b604f10bd489387f5583435414504: Neither blocks nor log segments found. Creating new log.
W20260810 07:27:45.473304 2196 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:27:45.473400 2196 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:27:45.473703 2145 log.cc:824] T 00000000000000000000000000000000 P 234b604f10bd489387f5583435414504: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:45.475891 2145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 234b604f10bd489387f5583435414504: No bootstrap required, opened a new log
I20260810 07:27:45.476177 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.460969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36538 (local address 127.12.157.253:40585)
0810 07:27:45.461081 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:45.461086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:45.461454 (+ 368us) server_negotiation.cc:407] Connection header received
0810 07:27:45.461547 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:45.461550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:45.461596 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:45.461672 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:45.462512 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.463214 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.464286 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.464596 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.464672 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:45.472391 (+ 7719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:45.472407 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:45.472410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:45.472436 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:45.474143 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:45.474695 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:45.474699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:45.474701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:45.474788 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:45.475101 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:45.475105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:45.475106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:45.475341 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:45.475515 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:45.475904 (+ 389us) server_negotiation.cc:299] Negotiation successful
0810 07:27:45.476034 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 07:27:45.473799 1922 master_runner.cc:186] Error getting master registration for 127.12.157.252:36451: OK, instance_id { permanent_uuid: "234b604f10bd489387f5583435414504" instance_seqno: 1786346865410631 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 07:27:45.477155 1922 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 07:27:45.479852 12919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:27:45.480301 2145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 234b604f10bd489387f5583435414504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "776bcd586cd24e9cac41005fb1037c13" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41661 } } peers { permanent_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 40585 } } peers { permanent_uuid: "234b604f10bd489387f5583435414504" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 36451 } }
I20260810 07:27:45.480532 2145 raft_consensus.cc:374] T 00000000000000000000000000000000 P 234b604f10bd489387f5583435414504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:45.480609 2145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 234b604f10bd489387f5583435414504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 234b604f10bd489387f5583435414504, State: Initialized, Role: FOLLOWER
I20260810 07:27:45.481176 2145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 234b604f10bd489387f5583435414504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "776bcd586cd24e9cac41005fb1037c13" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41661 } } peers { permanent_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 40585 } } peers { permanent_uuid: "234b604f10bd489387f5583435414504" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 36451 } }
I20260810 07:27:45.482810 2145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 234b604f10bd489387f5583435414504 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:27:45.481824 2204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 234b604f10bd489387f5583435414504 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "776bcd586cd24e9cac41005fb1037c13" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41661 } } peers { permanent_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 40585 } } peers { permanent_uuid: "234b604f10bd489387f5583435414504" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 36451 } } }
I20260810 07:27:45.483501 2204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 234b604f10bd489387f5583435414504 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 07:27:45.489285 2222 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 234b604f10bd489387f5583435414504: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:27:45.489441 2222 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:27:45.493901 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.483074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46528 (local address 127.12.157.254:41661)
0810 07:27:45.483202 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:45.483209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:45.483858 (+ 649us) server_negotiation.cc:407] Connection header received
0810 07:27:45.484101 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:45.484105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:45.484164 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:45.484220 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:45.485371 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.486150 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.487608 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.487936 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.488024 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:45.489994 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:45.489998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:45.490000 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:45.490028 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:45.491825 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:45.492326 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:45.492331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:45.492343 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:45.492439 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:45.492817 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:45.492824 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:45.492825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:45.493094 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:45.493225 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:45.493603 (+ 378us) server_negotiation.cc:299] Negotiation successful
0810 07:27:45.493725 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 07:27:45.495563 2038 master_runner.cc:263] No leader master found from master 234b604f10bd489387f5583435414504
I20260810 07:27:45.495630 2038 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 07:27:45.499445 2180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:27:45.499627 2180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "776bcd586cd24e9cac41005fb1037c13" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41661 } } peers { permanent_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 40585 } } peers { permanent_uuid: "234b604f10bd489387f5583435414504" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 36451 } }
I20260810 07:27:45.500671 2180 leader_election.cc:288] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 776bcd586cd24e9cac41005fb1037c13 (127.12.157.254:41661), 234b604f10bd489387f5583435414504 (127.12.157.252:36451)
I20260810 07:27:45.501505 2102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "234b604f10bd489387f5583435414504" is_pre_election: true
I20260810 07:27:45.501809 2102 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 234b604f10bd489387f5583435414504 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a029fb8f8034c5e8d1eb568f3773d2e in term 0.
I20260810 07:27:45.502218 1950 leader_election.cc:302] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 234b604f10bd489387f5583435414504, 7a029fb8f8034c5e8d1eb568f3773d2e; no voters:
I20260810 07:27:45.502909 1878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "776bcd586cd24e9cac41005fb1037c13" is_pre_election: true
I20260810 07:27:45.503093 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.483288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36546 (local address 127.12.157.253:40585)
0810 07:27:45.483397 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:45.483401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:45.491201 (+ 7800us) server_negotiation.cc:407] Connection header received
0810 07:27:45.491821 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:45.491825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:45.491884 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:45.491944 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:45.493232 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.493945 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.496270 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.496623 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.496722 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:45.498640 (+ 1918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:45.498646 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:45.498651 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:45.498683 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:45.501156 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:45.501667 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:45.501674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:45.501677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:45.501759 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:45.502076 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:45.502081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:45.502084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:45.502311 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:45.502451 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:45.502800 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 07:27:45.502917 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 07:27:45.503189 1878 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 776bcd586cd24e9cac41005fb1037c13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a029fb8f8034c5e8d1eb568f3773d2e in term 0.
I20260810 07:27:45.503307 2157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:45.493506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36562 (local address 127.12.157.252:36451)
0810 07:27:45.493611 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:45.493615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:45.494509 (+ 894us) server_negotiation.cc:407] Connection header received
0810 07:27:45.494680 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:45.494683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:45.494737 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:45.494792 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:45.495724 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.496466 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.497569 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:45.497815 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:45.497929 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:45.499462 (+ 1533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:45.499467 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:45.499469 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:45.499498 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:45.501291 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:45.501969 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:45.501973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:45.501975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:45.502063 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:45.502373 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:45.502376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:45.502378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:45.502593 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:45.502709 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:45.503060 (+ 351us) server_negotiation.cc:299] Negotiation successful
0810 07:27:45.503175 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 07:27:45.504673 2180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:27:45.504774 2180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:27:45.504832 2180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:45.506428 2180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "776bcd586cd24e9cac41005fb1037c13" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41661 } } peers { permanent_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 40585 } } peers { permanent_uuid: "234b604f10bd489387f5583435414504" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 36451 } }
I20260810 07:27:45.507148 2102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "234b604f10bd489387f5583435414504"
I20260810 07:27:45.507277 2102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 234b604f10bd489387f5583435414504 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:45.508635 2102 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 234b604f10bd489387f5583435414504 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a029fb8f8034c5e8d1eb568f3773d2e in term 1.
I20260810 07:27:45.508987 1950 leader_election.cc:302] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 234b604f10bd489387f5583435414504, 7a029fb8f8034c5e8d1eb568f3773d2e; no voters:
I20260810 07:27:45.509404 1878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "776bcd586cd24e9cac41005fb1037c13"
I20260810 07:27:45.509555 1878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 776bcd586cd24e9cac41005fb1037c13 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:45.510903 1878 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 776bcd586cd24e9cac41005fb1037c13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a029fb8f8034c5e8d1eb568f3773d2e in term 1.
I20260810 07:27:45.511745 2180 leader_election.cc:288] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e [CANDIDATE]: Term 1 election: Requested vote from peers 776bcd586cd24e9cac41005fb1037c13 (127.12.157.254:41661), 234b604f10bd489387f5583435414504 (127.12.157.252:36451)
I20260810 07:27:45.511897 2180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:27:45.511969 2180 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e [term 1 LEADER]: Becoming Leader. State: Replica: 7a029fb8f8034c5e8d1eb568f3773d2e, State: Running, Role: LEADER
I20260810 07:27:45.512307 2180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "776bcd586cd24e9cac41005fb1037c13" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41661 } } peers { permanent_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 40585 } } peers { permanent_uuid: "234b604f10bd489387f5583435414504" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 36451 } }
I20260810 07:27:45.514175 2180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a029fb8f8034c5e8d1eb568f3773d2e. Latest consensus state: current_term: 1 leader_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "776bcd586cd24e9cac41005fb1037c13" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41661 } } peers { permanent_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 40585 } } peers { permanent_uuid: "234b604f10bd489387f5583435414504" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 36451 } } }
I20260810 07:27:45.514276 2180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:45.514810 2231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:27:45.515815 2231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:27:45.522469 2102 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 234b604f10bd489387f5583435414504 [term 1 FOLLOWER]: Refusing update from remote peer 7a029fb8f8034c5e8d1eb568f3773d2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:27:45.523348 2180 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "234b604f10bd489387f5583435414504" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 36451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:27:45.524477 1878 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 776bcd586cd24e9cac41005fb1037c13 [term 1 FOLLOWER]: Refusing update from remote peer 7a029fb8f8034c5e8d1eb568f3773d2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:27:45.525089 2180 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "776bcd586cd24e9cac41005fb1037c13" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:27:45.528923 2204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 234b604f10bd489387f5583435414504 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a029fb8f8034c5e8d1eb568f3773d2e. Latest consensus state: current_term: 1 leader_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "776bcd586cd24e9cac41005fb1037c13" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41661 } } peers { permanent_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 40585 } } peers { permanent_uuid: "234b604f10bd489387f5583435414504" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 36451 } } }
I20260810 07:27:45.529062 2204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 234b604f10bd489387f5583435414504 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:27:45.530656 2161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 776bcd586cd24e9cac41005fb1037c13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a029fb8f8034c5e8d1eb568f3773d2e. Latest consensus state: current_term: 1 leader_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "776bcd586cd24e9cac41005fb1037c13" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41661 } } peers { permanent_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 40585 } } peers { permanent_uuid: "234b604f10bd489387f5583435414504" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 36451 } } }
I20260810 07:27:45.530769 2161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 776bcd586cd24e9cac41005fb1037c13 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:27:45.536207 2180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "776bcd586cd24e9cac41005fb1037c13" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41661 } } peers { permanent_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 40585 } } peers { permanent_uuid: "234b604f10bd489387f5583435414504" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 36451 } } }
I20260810 07:27:45.536327 2180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:45.539947 2161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 776bcd586cd24e9cac41005fb1037c13 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "776bcd586cd24e9cac41005fb1037c13" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41661 } } peers { permanent_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 40585 } } peers { permanent_uuid: "234b604f10bd489387f5583435414504" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 36451 } } }
I20260810 07:27:45.540058 2161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 776bcd586cd24e9cac41005fb1037c13 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:27:45.540546 2232 mvcc.cc:205] Tried to move back new op lower bound from 7316876761172299776 to 7316876761141719040. Current Snapshot: MvccSnapshot[applied={T|T < 7316876761172299776}]
I20260810 07:27:45.540999 2239 mvcc.cc:205] Tried to move back new op lower bound from 7316876761172299776 to 7316876761141719040. Current Snapshot: MvccSnapshot[applied={T|T < 7316876761172299776}]
I20260810 07:27:45.542698 2231 catalog_manager.cc:1347] Generated new cluster ID: 02d81e38c4e24ace99d5387ec1b2e39e
I20260810 07:27:45.542758 2231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:27:45.545706 2180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "776bcd586cd24e9cac41005fb1037c13" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41661 } } peers { permanent_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 40585 } } peers { permanent_uuid: "234b604f10bd489387f5583435414504" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 36451 } } }
I20260810 07:27:45.545804 2180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:45.547518 2204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 234b604f10bd489387f5583435414504 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "776bcd586cd24e9cac41005fb1037c13" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41661 } } peers { permanent_uuid: "7a029fb8f8034c5e8d1eb568f3773d2e" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 40585 } } peers { permanent_uuid: "234b604f10bd489387f5583435414504" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 36451 } } }
I20260810 07:27:45.547617 2204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 234b604f10bd489387f5583435414504 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:27:45.548482 2233 mvcc.cc:205] Tried to move back new op lower bound from 7316876761172299776 to 7316876761141719040. Current Snapshot: MvccSnapshot[applied={T|T < 7316876761172299776}]
I20260810 07:27:45.578179 2231 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:27:45.579586 2231 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:27:45.591531 2231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a029fb8f8034c5e8d1eb568f3773d2e: Generated new TSK 0
I20260810 07:27:45.593091 2231 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:27:46.471235 2178 catalog_manager.cc:1259] Loaded cluster ID: 02d81e38c4e24ace99d5387ec1b2e39e
I20260810 07:27:46.471323 2178 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 776bcd586cd24e9cac41005fb1037c13: loading cluster ID for follower catalog manager: success
I20260810 07:27:46.473006 2178 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 776bcd586cd24e9cac41005fb1037c13: acquiring CA information for follower catalog manager: success
I20260810 07:27:46.474135 2178 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 776bcd586cd24e9cac41005fb1037c13: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 07:27:46.490864 2222 catalog_manager.cc:1259] Loaded cluster ID: 02d81e38c4e24ace99d5387ec1b2e39e
I20260810 07:27:46.490969 2222 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 234b604f10bd489387f5583435414504: loading cluster ID for follower catalog manager: success
I20260810 07:27:46.492635 2222 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 234b604f10bd489387f5583435414504: acquiring CA information for follower catalog manager: success
I20260810 07:27:46.493814 2222 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 234b604f10bd489387f5583435414504: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 07:27:46.664669 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 1811
I20260810 07:27:46.679481 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 1922
I20260810 07:27:46.692219 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 2038
2026-08-10T07:27:46Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3355 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:27:46 dist-test-slave-ffrd krb5kdc[2257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:27:46 dist-test-slave-ffrd krb5kdc[2257](info): set up 2 sockets
Aug 10 07:27:46 dist-test-slave-ffrd krb5kdc[2257](info): commencing operation
krb5kdc: starting...
W20260810 07:27:47.974480 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.254s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:27:48 dist-test-slave-ffrd krb5kdc[2257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:27:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:27:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:27:48.106062 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:40949
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:40691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:40949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:48.226269 2279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:48.226649 2279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:48.226737 2279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:48.231249 2279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:27:48.231575 2279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:48.231805 2279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:48.231853 2279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:27:48.231900 2279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:27:48.239151 2279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:40691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:40949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:40949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:27:48.240409 2279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:48.241539 2279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:48.250300 2299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:48.250304 2290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:48.250591 2279 server_base.cc:1034] running on GCE node
W20260810 07:27:48.250304 2288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:48.251351 2279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:48.252503 2279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:48.253710 2279 hybrid_clock.cc:648] HybridClock initialized: now 1786346868253683 us; error 52 us; skew 500 ppm
Aug 10 07:27:48 dist-test-slave-ffrd krb5kdc[2257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346868, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:48.256690 2279 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:27:48.257690 2279 webserver.cc:460] Webserver started at http://127.12.157.254:38147/ using document root <none> and password file <none>
I20260810 07:27:48.258275 2279 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:48.258370 2279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:48.258613 2279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:27:48.260708 2279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "f2dd3bcb75c946999238f04ef06efb36"
format_stamp: "Formatted at 2026-08-10 07:27:48 on dist-test-slave-ffrd"
server_key: "944758b2930153a11ac68d8e772bfb09"
server_key_iv: "9a8eaf30b8fd9d487e4a1863262b80c7"
server_key_version: "encryptionkey@0"
I20260810 07:27:48.261420 2279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "f2dd3bcb75c946999238f04ef06efb36"
format_stamp: "Formatted at 2026-08-10 07:27:48 on dist-test-slave-ffrd"
server_key: "944758b2930153a11ac68d8e772bfb09"
server_key_iv: "9a8eaf30b8fd9d487e4a1863262b80c7"
server_key_version: "encryptionkey@0"
I20260810 07:27:48.265280 2279 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 07:27:48.267756 2308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:48.269073 2279 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:27:48.269210 2279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "f2dd3bcb75c946999238f04ef06efb36"
format_stamp: "Formatted at 2026-08-10 07:27:48 on dist-test-slave-ffrd"
server_key: "944758b2930153a11ac68d8e772bfb09"
server_key_iv: "9a8eaf30b8fd9d487e4a1863262b80c7"
server_key_version: "encryptionkey@0"
I20260810 07:27:48.269326 2279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:48.287774 2279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:48.291087 2279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:48.291326 2279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:48.298728 2279 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:40949
I20260810 07:27:48.298745 2383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:40949 every 8 connection(s)
I20260810 07:27:48.299742 2279 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:27:48.302723 2386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:48.303938 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 2279
I20260810 07:27:48.304061 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:27:48.304368 12919 external_mini_cluster.cc:1428] Setting key be6d7298b92b798b30eca7a45d01d123
I20260810 07:27:48.309074 2386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2dd3bcb75c946999238f04ef06efb36: Bootstrap starting.
I20260810 07:27:48.311268 2386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2dd3bcb75c946999238f04ef06efb36: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:48.312153 2386 log.cc:824] T 00000000000000000000000000000000 P f2dd3bcb75c946999238f04ef06efb36: Log is configured to *not* fsync() on all Append() calls
Aug 10 07:27:48 dist-test-slave-ffrd krb5kdc[2257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346868, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:27:48.314145 2386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2dd3bcb75c946999238f04ef06efb36: No bootstrap required, opened a new log
I20260810 07:27:48.317766 2386 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2dd3bcb75c946999238f04ef06efb36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2dd3bcb75c946999238f04ef06efb36" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 40949 } }
I20260810 07:27:48.318001 2386 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2dd3bcb75c946999238f04ef06efb36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:48.318060 2386 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2dd3bcb75c946999238f04ef06efb36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2dd3bcb75c946999238f04ef06efb36, State: Initialized, Role: FOLLOWER
I20260810 07:27:48.318526 2389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:48.305798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51044 (local address 127.12.157.254:40949)
0810 07:27:48.306286 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:48.306303 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:48.306358 (+ 55us) server_negotiation.cc:407] Connection header received
0810 07:27:48.307058 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:48.307076 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:48.307415 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:48.307757 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:48.308586 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:48.309670 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:48.310536 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:48.310993 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:48.311167 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:48.313796 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:48.313801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:48.313815 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:48.313841 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:48.315483 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:48.315937 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:48.315942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:48.315946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:48.316056 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:48.316340 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:48.316344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:48.316346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:48.316715 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:48.316970 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:48.317223 (+ 253us) server_negotiation.cc:299] Negotiation successful
0810 07:27:48.317512 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":139,"threads_started":1}
I20260810 07:27:48.318490 2386 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2dd3bcb75c946999238f04ef06efb36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2dd3bcb75c946999238f04ef06efb36" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 40949 } }
I20260810 07:27:48.318699 2386 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2dd3bcb75c946999238f04ef06efb36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:27:48.318796 2386 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2dd3bcb75c946999238f04ef06efb36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:27:48.318912 2386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2dd3bcb75c946999238f04ef06efb36 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:48.319937 2386 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2dd3bcb75c946999238f04ef06efb36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2dd3bcb75c946999238f04ef06efb36" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 40949 } }
I20260810 07:27:48.320290 2386 leader_election.cc:302] T 00000000000000000000000000000000 P f2dd3bcb75c946999238f04ef06efb36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2dd3bcb75c946999238f04ef06efb36; no voters:
I20260810 07:27:48.320713 2386 leader_election.cc:288] T 00000000000000000000000000000000 P f2dd3bcb75c946999238f04ef06efb36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:27:48.320783 2393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2dd3bcb75c946999238f04ef06efb36 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:27:48.321058 2393 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2dd3bcb75c946999238f04ef06efb36 [term 1 LEADER]: Becoming Leader. State: Replica: f2dd3bcb75c946999238f04ef06efb36, State: Running, Role: LEADER
I20260810 07:27:48.321410 2393 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2dd3bcb75c946999238f04ef06efb36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2dd3bcb75c946999238f04ef06efb36" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 40949 } }
I20260810 07:27:48.321658 12919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:27:48.322021 2386 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2dd3bcb75c946999238f04ef06efb36 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:27:48.322652 2393 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2dd3bcb75c946999238f04ef06efb36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2dd3bcb75c946999238f04ef06efb36. Latest consensus state: current_term: 1 leader_uuid: "f2dd3bcb75c946999238f04ef06efb36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2dd3bcb75c946999238f04ef06efb36" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 40949 } } }
I20260810 07:27:48.322775 2393 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2dd3bcb75c946999238f04ef06efb36 [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:48.323144 2394 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2dd3bcb75c946999238f04ef06efb36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2dd3bcb75c946999238f04ef06efb36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2dd3bcb75c946999238f04ef06efb36" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 40949 } } }
I20260810 07:27:48.323256 2394 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2dd3bcb75c946999238f04ef06efb36 [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:48.323493 2401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:27:48.326912 2401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:27:48.332620 2401 catalog_manager.cc:1347] Generated new cluster ID: 1168b7bd037b4ad29737019fff30bc75
I20260810 07:27:48.332715 2401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:27:48.348794 2401 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:27:48.350104 2401 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:27:48.361060 2401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2dd3bcb75c946999238f04ef06efb36: Generated new TSK 0
I20260810 07:27:48.361881 2401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:27:48.377573 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 2279
2026-08-10T07:27:48Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1686 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:27:48 dist-test-slave-ffrd krb5kdc[2426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:27:48 dist-test-slave-ffrd krb5kdc[2426](info): set up 2 sockets
Aug 10 07:27:48 dist-test-slave-ffrd krb5kdc[2426](info): commencing operation
krb5kdc: starting...
W20260810 07:27:49.670111 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.263s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:27:49 dist-test-slave-ffrd krb5kdc[2426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:27:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:27:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:27:49.803907 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:39709
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:46101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:39709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:49.920645 2440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:49.920989 2440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:49.921099 2440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:49.925091 2440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:27:49.925182 2440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:49.925222 2440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:49.925259 2440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:27:49.925290 2440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:27:49.930610 2440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:46101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:39709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:39709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:27:49.931672 2440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:49.932698 2440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:49.941372 2449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:49.941421 2450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:49.941375 2455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:49.941682 2440 server_base.cc:1034] running on GCE node
I20260810 07:27:49.942247 2440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:49.943405 2440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:49.944595 2440 hybrid_clock.cc:648] HybridClock initialized: now 1786346869944577 us; error 31 us; skew 500 ppm
Aug 10 07:27:49 dist-test-slave-ffrd krb5kdc[2426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346869, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:49.947561 2440 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:27:49.948474 2440 webserver.cc:460] Webserver started at http://127.12.157.254:41799/ using document root <none> and password file <none>
I20260810 07:27:49.949201 2440 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:49.949286 2440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:49.949481 2440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:27:49.951512 2440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "c72a6929623c409db8ae2e39c8aacf9a"
format_stamp: "Formatted at 2026-08-10 07:27:49 on dist-test-slave-ffrd"
server_key: "bf0dd917acaa08a98ffde4343e253ad4"
server_key_iv: "95f337f7d79b3b8a200c523842766a67"
server_key_version: "encryptionkey@0"
I20260810 07:27:49.952030 2440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "c72a6929623c409db8ae2e39c8aacf9a"
format_stamp: "Formatted at 2026-08-10 07:27:49 on dist-test-slave-ffrd"
server_key: "bf0dd917acaa08a98ffde4343e253ad4"
server_key_iv: "95f337f7d79b3b8a200c523842766a67"
server_key_version: "encryptionkey@0"
I20260810 07:27:49.955734 2440 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 07:27:49.958069 2468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:49.959311 2440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 07:27:49.959455 2440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "c72a6929623c409db8ae2e39c8aacf9a"
format_stamp: "Formatted at 2026-08-10 07:27:49 on dist-test-slave-ffrd"
server_key: "bf0dd917acaa08a98ffde4343e253ad4"
server_key_iv: "95f337f7d79b3b8a200c523842766a67"
server_key_version: "encryptionkey@0"
I20260810 07:27:49.959563 2440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:49.997402 2440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:50.000913 2440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:50.001163 2440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:50.009143 2440 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:39709
I20260810 07:27:50.009474 2541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:39709 every 8 connection(s)
I20260810 07:27:50.010226 2440 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:27:50.012040 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 2440
I20260810 07:27:50.012173 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:27:50.012455 12919 external_mini_cluster.cc:1428] Setting key 9527f33d86802283a5d7ce1e140f10fe
I20260810 07:27:50.013731 2542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:50.020557 2542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a: Bootstrap starting.
Aug 10 07:27:50 dist-test-slave-ffrd krb5kdc[2426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:27:50.023383 2542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:50.024328 2542 log.cc:824] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:50.026466 2542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a: No bootstrap required, opened a new log
I20260810 07:27:50.028471 2548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:50.014020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47226 (local address 127.12.157.254:39709)
0810 07:27:50.014475 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:50.014492 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:50.014530 (+ 38us) server_negotiation.cc:407] Connection header received
0810 07:27:50.015189 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:50.015204 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:50.015540 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:50.015817 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:50.016779 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:50.017946 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:50.019478 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:50.019963 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:50.020087 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:50.023181 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:50.023186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:50.023199 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:50.023231 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:50.025481 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:50.025965 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:50.025971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:50.025976 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:50.026090 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:50.026399 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:50.026402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:50.026404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:50.026872 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:50.027082 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:50.027422 (+ 340us) server_negotiation.cc:299] Negotiation successful
0810 07:27:50.027748 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":145,"threads_started":1}
I20260810 07:27:50.030141 2542 raft_consensus.cc:348] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72a6929623c409db8ae2e39c8aacf9a" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39709 } }
I20260810 07:27:50.030435 2542 raft_consensus.cc:374] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:50.030486 2542 raft_consensus.cc:729] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c72a6929623c409db8ae2e39c8aacf9a, State: Initialized, Role: FOLLOWER
I20260810 07:27:50.030964 2542 consensus_queue.cc:260] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72a6929623c409db8ae2e39c8aacf9a" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39709 } }
I20260810 07:27:50.031138 2542 raft_consensus.cc:388] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:27:50.031195 2542 raft_consensus.cc:482] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:27:50.031271 2542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:50.032181 2542 raft_consensus.cc:504] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72a6929623c409db8ae2e39c8aacf9a" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39709 } }
I20260810 07:27:50.032483 2542 leader_election.cc:302] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c72a6929623c409db8ae2e39c8aacf9a; no voters:
I20260810 07:27:50.032788 2542 leader_election.cc:288] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:27:50.032933 2553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:27:50.033250 2553 raft_consensus.cc:686] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [term 1 LEADER]: Becoming Leader. State: Replica: c72a6929623c409db8ae2e39c8aacf9a, State: Running, Role: LEADER
I20260810 07:27:50.033627 2553 consensus_queue.cc:237] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72a6929623c409db8ae2e39c8aacf9a" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39709 } }
I20260810 07:27:50.034145 2542 sys_catalog.cc:564] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:27:50.035367 2559 sys_catalog.cc:455] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c72a6929623c409db8ae2e39c8aacf9a. Latest consensus state: current_term: 1 leader_uuid: "c72a6929623c409db8ae2e39c8aacf9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72a6929623c409db8ae2e39c8aacf9a" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39709 } } }
I20260810 07:27:50.035513 2559 sys_catalog.cc:458] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:50.035923 2566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:27:50.035883 2554 sys_catalog.cc:455] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c72a6929623c409db8ae2e39c8aacf9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72a6929623c409db8ae2e39c8aacf9a" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39709 } } }
I20260810 07:27:50.036042 2554 sys_catalog.cc:458] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:50.039077 2566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:27:50.044662 2566 catalog_manager.cc:1347] Generated new cluster ID: d721cb79fc0641e0b89ce9aae9674901
I20260810 07:27:50.044749 2566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:27:50.056173 2566 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:27:50.057507 2566 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:27:50.063195 2566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a: Generated new TSK 0
I20260810 07:27:50.063947 2566 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
I20260810 07:27:50.099723 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:0
--local_ip_for_outbound_sockets=127.12.157.193
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:39709
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:46101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:50.246186 2588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:50.246476 2588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:50.246557 2588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:50.250823 2588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:50.250891 2588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:50.250993 2588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:27:50.256364 2588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:46101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:39709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:27:50.257550 2588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:50.258669 2588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:50.268854 2594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:50.268860 2595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:50.268900 2599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:50.269203 2588 server_base.cc:1034] running on GCE node
I20260810 07:27:50.269761 2588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:50.270378 2588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:50.271565 2588 hybrid_clock.cc:648] HybridClock initialized: now 1786346870271544 us; error 30 us; skew 500 ppm
Aug 10 07:27:50 dist-test-slave-ffrd krb5kdc[2426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346870, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:50.274598 2588 init.cc:377] Logged in from keytab as kudu/127.12.157.193@KRBTEST.COM (short username kudu)
I20260810 07:27:50.275564 2588 webserver.cc:460] Webserver started at http://127.12.157.193:46173/ using document root <none> and password file <none>
I20260810 07:27:50.276154 2588 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:50.276218 2588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:50.276448 2588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:27:50.278679 2588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/data/instance:
uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769"
format_stamp: "Formatted at 2026-08-10 07:27:50 on dist-test-slave-ffrd"
server_key: "b6b72b66603fd0e3ccedf473e1313806"
server_key_iv: "58712e49785086b3d04a96752a52aca4"
server_key_version: "encryptionkey@0"
I20260810 07:27:50.279253 2588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance:
uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769"
format_stamp: "Formatted at 2026-08-10 07:27:50 on dist-test-slave-ffrd"
server_key: "b6b72b66603fd0e3ccedf473e1313806"
server_key_iv: "58712e49785086b3d04a96752a52aca4"
server_key_version: "encryptionkey@0"
I20260810 07:27:50.283135 2588 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 07:27:50.285560 2614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:50.286746 2588 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:27:50.286888 2588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769"
format_stamp: "Formatted at 2026-08-10 07:27:50 on dist-test-slave-ffrd"
server_key: "b6b72b66603fd0e3ccedf473e1313806"
server_key_iv: "58712e49785086b3d04a96752a52aca4"
server_key_version: "encryptionkey@0"
I20260810 07:27:50.287001 2588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:50.318399 2588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:50.321902 2588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:50.322118 2588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:50.322745 2588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:50.323621 2588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:50.323711 2588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:50.323786 2588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:50.323823 2588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:50.333289 2588 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:41497
I20260810 07:27:50.333304 2808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:41497 every 8 connection(s)
I20260810 07:27:50.334260 2588 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:27:50.339210 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 2588
I20260810 07:27:50.339371 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance
I20260810 07:27:50.339638 12919 external_mini_cluster.cc:1428] Setting key 9c9d014c4a15fac9e6c7de59cb1b122c
Aug 10 07:27:50 dist-test-slave-ffrd krb5kdc[2426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346870, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:27:50.349164 2548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:50.336220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:47791 (local address 127.12.157.254:39709)
0810 07:27:50.336386 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:50.336391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:50.337312 (+ 921us) server_negotiation.cc:407] Connection header received
0810 07:27:50.338664 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:50.338667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:50.338716 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:50.338796 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:50.340081 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:50.340746 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:50.342013 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:50.342425 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:50.342554 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:50.345394 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:50.345398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:50.345400 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:50.345425 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:50.346961 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:50.347532 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:50.347535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:50.347537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:50.347614 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:50.347984 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:50.347987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:50.347989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:50.348177 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:50.348292 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:50.348799 (+ 507us) server_negotiation.cc:299] Negotiation successful
0810 07:27:50.348982 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 07:27:50.349951 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:50.336636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.193:47791)
0810 07:27:50.337158 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:50.337201 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:50.338468 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:50.338970 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:50.338976 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:50.339370 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:50.339934 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:50.339948 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:50.340905 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:50.340909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:50.341877 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:50.341888 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:50.342600 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:50.342603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:50.342750 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:50.343430 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:50.343455 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:50.345217 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:50.347111 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:50.347118 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:50.347133 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:50.347406 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:50.347715 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:50.347719 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:50.347721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:50.347878 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:50.348293 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:50.348300 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:50.348557 (+ 257us) client_negotiation.cc:769] Sending connection context
0810 07:27:50.348741 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 07:27:50.349109 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":128,"threads_started":1}
WARNING: no policy specified for kudu/127.12.157.194@KRBTEST.COM; defaulting to no policy
I20260810 07:27:50.351188 2809 heartbeater.cc:347] Connected to a master server at 127.12.157.254:39709
I20260810 07:27:50.351447 2809 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:50.352005 2809 heartbeater.cc:511] Master 127.12.157.254:39709 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.194@KRBTEST.COM" created.
I20260810 07:27:50.353731 2492 ts_manager.cc:194] Registered new tserver with Master: dcf4b29ce0ac4dcc97cf34daed3fe769 (127.12.157.193:41497)
I20260810 07:27:50.354869 2492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.193@KRBTEST.COM'} at 127.12.157.193:47791
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
I20260810 07:27:50.370110 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:0
--local_ip_for_outbound_sockets=127.12.157.194
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:39709
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:46101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:50.488999 2816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:50.489280 2816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:50.489326 2816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:50.493939 2816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:50.494009 2816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:50.494107 2816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:27:50.499516 2816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:46101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:39709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:27:50.500627 2816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:50.501710 2816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:50.512426 2831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:50.512449 2816 server_base.cc:1034] running on GCE node
W20260810 07:27:50.512409 2825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:50.512409 2823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:50.513123 2816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:50.513808 2816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:50.515021 2816 hybrid_clock.cc:648] HybridClock initialized: now 1786346870514996 us; error 36 us; skew 500 ppm
Aug 10 07:27:50 dist-test-slave-ffrd krb5kdc[2426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346870, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:50.518167 2816 init.cc:377] Logged in from keytab as kudu/127.12.157.194@KRBTEST.COM (short username kudu)
I20260810 07:27:50.519081 2816 webserver.cc:460] Webserver started at http://127.12.157.194:35389/ using document root <none> and password file <none>
I20260810 07:27:50.519667 2816 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:50.519771 2816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:50.520020 2816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:27:50.522156 2816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/data/instance:
uuid: "f4c5fa460e5a4a648bda699e263e0940"
format_stamp: "Formatted at 2026-08-10 07:27:50 on dist-test-slave-ffrd"
server_key: "c0397f75dfbe44945d29906535c8fb88"
server_key_iv: "31ca7d568640b81076c0f66897d7ffbc"
server_key_version: "encryptionkey@0"
I20260810 07:27:50.522748 2816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance:
uuid: "f4c5fa460e5a4a648bda699e263e0940"
format_stamp: "Formatted at 2026-08-10 07:27:50 on dist-test-slave-ffrd"
server_key: "c0397f75dfbe44945d29906535c8fb88"
server_key_iv: "31ca7d568640b81076c0f66897d7ffbc"
server_key_version: "encryptionkey@0"
I20260810 07:27:50.526715 2816 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 07:27:50.529464 2842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:50.530755 2816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:27:50.530905 2816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "f4c5fa460e5a4a648bda699e263e0940"
format_stamp: "Formatted at 2026-08-10 07:27:50 on dist-test-slave-ffrd"
server_key: "c0397f75dfbe44945d29906535c8fb88"
server_key_iv: "31ca7d568640b81076c0f66897d7ffbc"
server_key_version: "encryptionkey@0"
I20260810 07:27:50.531044 2816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:50.546603 2816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:50.549826 2816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:50.550051 2816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:50.550676 2816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:50.551544 2816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:50.551620 2816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:50.551692 2816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:50.551743 2816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:50.561123 2816 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:36057
I20260810 07:27:50.561146 3002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:36057 every 8 connection(s)
I20260810 07:27:50.562052 2816 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:27:50.567981 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 2816
I20260810 07:27:50.568127 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance
I20260810 07:27:50.568425 12919 external_mini_cluster.cc:1428] Setting key ea13555ff5946ebe7703ba4f1fe2d1a2
Aug 10 07:27:50 dist-test-slave-ffrd krb5kdc[2426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346870, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:27:50.578516 2548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:50.564053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:57259 (local address 127.12.157.254:39709)
0810 07:27:50.564196 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:50.564199 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:50.565142 (+ 943us) server_negotiation.cc:407] Connection header received
0810 07:27:50.566618 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:50.566622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:50.566672 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:50.566770 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:50.568145 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:50.568831 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:50.570421 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:50.570648 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:50.570728 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:50.573868 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:50.573875 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:50.573881 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:50.573922 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:50.575710 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:50.576477 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:50.576481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:50.576483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:50.576590 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:50.577077 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:50.577079 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:50.577080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:50.577319 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:50.577403 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:50.578232 (+ 829us) server_negotiation.cc:299] Negotiation successful
0810 07:27:50.578374 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 07:27:50.579879 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:50.564467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.194:57259)
0810 07:27:50.564976 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:50.565032 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:50.566391 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:50.567067 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:50.567074 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:50.567463 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:50.567978 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:50.567992 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:50.569029 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:50.569033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:50.570270 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:50.570285 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:50.570806 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:50.570813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:50.571072 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:50.571782 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:50.571809 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:50.573648 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:50.575895 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:50.575901 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:50.575916 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:50.576327 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:50.576701 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:50.576704 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:50.576705 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:50.576895 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:50.577446 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:50.577457 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:50.577895 (+ 438us) client_negotiation.cc:769] Sending connection context
0810 07:27:50.578171 (+ 276us) client_negotiation.cc:240] Negotiation successful
0810 07:27:50.578654 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":140,"threads_started":1}
WARNING: no policy specified for kudu/127.12.157.195@KRBTEST.COM; defaulting to no policy
I20260810 07:27:50.581274 3003 heartbeater.cc:347] Connected to a master server at 127.12.157.254:39709
I20260810 07:27:50.581590 3003 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:50.582095 3003 heartbeater.cc:511] Master 127.12.157.254:39709 requested a full tablet report, sending...
I20260810 07:27:50.583289 2492 ts_manager.cc:194] Registered new tserver with Master: f4c5fa460e5a4a648bda699e263e0940 (127.12.157.194:36057)
I20260810 07:27:50.583976 2492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.194@KRBTEST.COM'} at 127.12.157.194:57259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
I20260810 07:27:50.604347 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:0
--local_ip_for_outbound_sockets=127.12.157.195
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:39709
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:46101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:50.726485 3010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:50.726763 3010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:50.726807 3010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:50.730796 3010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:50.730865 3010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:50.730926 3010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:27:50.736755 3010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:46101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:39709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:27:50.737900 3010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:50.738893 3010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:50.749325 3019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:50.749361 3018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:50.749495 3010 server_base.cc:1034] running on GCE node
W20260810 07:27:50.749775 3022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:50.750267 3010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:50.750982 3010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:50.752189 3010 hybrid_clock.cc:648] HybridClock initialized: now 1786346870752171 us; error 28 us; skew 500 ppm
Aug 10 07:27:50 dist-test-slave-ffrd krb5kdc[2426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346870, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:50.755331 3010 init.cc:377] Logged in from keytab as kudu/127.12.157.195@KRBTEST.COM (short username kudu)
I20260810 07:27:50.756300 3010 webserver.cc:460] Webserver started at http://127.12.157.195:46713/ using document root <none> and password file <none>
I20260810 07:27:50.756922 3010 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:50.757076 3010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:50.757326 3010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:27:50.759529 3010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/data/instance:
uuid: "5ba31ad84ad84a7eb41bae121def8fc6"
format_stamp: "Formatted at 2026-08-10 07:27:50 on dist-test-slave-ffrd"
server_key: "8d14b162bd5a5b6a15b81be775336175"
server_key_iv: "72e1b38a26936d3b2ec5a90c620d2883"
server_key_version: "encryptionkey@0"
I20260810 07:27:50.760133 3010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance:
uuid: "5ba31ad84ad84a7eb41bae121def8fc6"
format_stamp: "Formatted at 2026-08-10 07:27:50 on dist-test-slave-ffrd"
server_key: "8d14b162bd5a5b6a15b81be775336175"
server_key_iv: "72e1b38a26936d3b2ec5a90c620d2883"
server_key_version: "encryptionkey@0"
I20260810 07:27:50.764376 3010 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 07:27:50.767168 3032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:50.768579 3010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 07:27:50.768750 3010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "5ba31ad84ad84a7eb41bae121def8fc6"
format_stamp: "Formatted at 2026-08-10 07:27:50 on dist-test-slave-ffrd"
server_key: "8d14b162bd5a5b6a15b81be775336175"
server_key_iv: "72e1b38a26936d3b2ec5a90c620d2883"
server_key_version: "encryptionkey@0"
I20260810 07:27:50.768889 3010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:50.784776 3010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:50.788280 3010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:50.788520 3010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:50.789223 3010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:50.790109 3010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:50.790182 3010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:50.790256 3010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:50.790309 3010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:50.799964 3010 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:42671
I20260810 07:27:50.799994 3194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:42671 every 8 connection(s)
I20260810 07:27:50.800866 3010 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:27:50.803457 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 3010
I20260810 07:27:50.803586 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance
I20260810 07:27:50.803925 12919 external_mini_cluster.cc:1428] Setting key a73e9b48977071403f9231cd5f194b5f
Aug 10 07:27:50 dist-test-slave-ffrd krb5kdc[2426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346870, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:27:50.815150 2548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:50.802801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:52423 (local address 127.12.157.254:39709)
0810 07:27:50.802945 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:50.802954 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:50.803778 (+ 824us) server_negotiation.cc:407] Connection header received
0810 07:27:50.805663 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:50.805666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:50.805727 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:50.805811 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:50.807064 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:50.807613 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:50.808778 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:50.809078 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:50.809190 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:50.811646 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:50.811650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:50.811653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:50.811679 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:50.813017 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:50.813569 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:50.813574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:50.813575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:50.813669 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:50.814024 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:50.814029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:50.814030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:50.814220 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:50.814329 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:50.814843 (+ 514us) server_negotiation.cc:299] Negotiation successful
0810 07:27:50.814975 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 07:27:50.815941 3198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:50.803108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.195:52423)
0810 07:27:50.803605 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:50.803658 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:50.805430 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:50.806008 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:50.806017 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:50.806404 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:50.806899 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:50.806913 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:50.807747 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:50.807750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:50.808646 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:50.808656 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:50.809233 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:50.809237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:50.809423 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:50.809980 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:50.809997 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:50.811479 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:50.813145 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:50.813151 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:50.813166 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:50.813458 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:50.813772 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:50.813775 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:50.813777 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:50.813921 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:50.814330 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:50.814338 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:50.814600 (+ 262us) client_negotiation.cc:769] Sending connection context
0810 07:27:50.814802 (+ 202us) client_negotiation.cc:240] Negotiation successful
0810 07:27:50.815132 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":135,"threads_started":1}
I20260810 07:27:50.817308 3196 heartbeater.cc:347] Connected to a master server at 127.12.157.254:39709
I20260810 07:27:50.817566 3196 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:50.818159 3196 heartbeater.cc:511] Master 127.12.157.254:39709 requested a full tablet report, sending...
I20260810 07:27:50.819166 2492 ts_manager.cc:194] Registered new tserver with Master: 5ba31ad84ad84a7eb41bae121def8fc6 (127.12.157.195:42671)
I20260810 07:27:50.819756 2492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.195@KRBTEST.COM'} at 127.12.157.195:52423
I20260810 07:27:50.829531 12919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:27:50.838979 2548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:50.831559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47242 (local address 127.12.157.254:39709)
0810 07:27:50.831680 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:50.831683 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:50.831884 (+ 201us) server_negotiation.cc:407] Connection header received
0810 07:27:50.831969 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:50.831971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:50.832007 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:50.832072 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:50.832717 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:50.833282 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:50.834107 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:50.834342 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:50.834438 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:50.836155 (+ 1717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:50.836158 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:50.836160 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:50.836181 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:50.837573 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:50.837944 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:50.837947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:50.837948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:50.838018 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:50.838243 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:50.838246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:50.838247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:50.838456 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:50.838556 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:50.838772 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 07:27:50.838866 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 07:27:50.841817 2492 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:27:50.844885 2492 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:27:50.860015 3210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:50.853664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:36057 (local address 127.0.0.1:39678)
0810 07:27:50.854749 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:50.854801 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:50.855017 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:50.855751 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:50.855756 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:50.855775 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:50.855879 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:50.855915 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:50.857169 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:50.857175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:50.858414 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:50.858424 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:50.859407 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:50.859411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:50.859522 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:50.859567 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 07:27:50.859695 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 07:27:50.859829 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":4,"thread_start_us":110,"threads_started":1}
I20260810 07:27:50.860031 3211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:50.853800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39678 (local address 127.12.157.194:36057)
0810 07:27:50.854491 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:50.854505 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:50.854897 (+ 392us) server_negotiation.cc:407] Connection header received
0810 07:27:50.855226 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:50.855235 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:50.855464 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:50.855569 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:50.856047 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:50.857032 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:50.858540 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:50.859269 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:50.859403 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:50.859558 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:50.859740 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 07:27:50.859855 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":132,"threads_started":1}
I20260810 07:27:50.860381 3215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:50.853827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:42671 (local address 127.0.0.1:48044)
0810 07:27:50.855175 (+ 1348us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:50.855189 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:50.855296 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:50.856217 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:50.856220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:50.856237 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:50.856323 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:50.856329 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:50.857723 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:50.857726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:50.858853 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:50.858864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:50.859960 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:50.859964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:50.860064 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:50.860094 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 07:27:50.860168 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 07:27:50.860239 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1284,"thread_start_us":127,"threads_started":1}
I20260810 07:27:50.860858 3213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:50.854048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48044 (local address 127.12.157.195:42671)
0810 07:27:50.854728 (+ 680us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:50.854742 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:50.855400 (+ 658us) server_negotiation.cc:407] Connection header received
0810 07:27:50.855488 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:50.855493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:50.855804 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:50.855959 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:50.856451 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:50.857582 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:50.858964 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:50.859665 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:50.859780 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:50.859894 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:50.860592 (+ 698us) server_negotiation.cc:299] Negotiation successful
0810 07:27:50.860695 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":96,"threads_started":1}
I20260810 07:27:50.860968 3212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:50.854800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:41497 (local address 127.0.0.1:37782)
0810 07:27:50.855188 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:50.855201 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:50.855278 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:50.855949 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:50.855963 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:50.855985 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:50.856092 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:50.856102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:50.857540 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:50.857544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:50.858781 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:50.858792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:50.860054 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:50.860061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:50.860648 (+ 587us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:50.860672 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 07:27:50.860748 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 07:27:50.860816 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":108,"threads_started":1}
I20260810 07:27:50.861370 3218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:50.854996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37782 (local address 127.12.157.193:41497)
0810 07:27:50.855545 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:50.855551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:50.855571 (+ 20us) server_negotiation.cc:407] Connection header received
0810 07:27:50.855637 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:50.855641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:50.855803 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:50.855952 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:50.856207 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:50.857379 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:50.858919 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:50.859733 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:50.859852 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:50.859981 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:50.860797 (+ 816us) server_negotiation.cc:299] Negotiation successful
0810 07:27:50.861216 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":98,"threads_started":1}
I20260810 07:27:50.863540 3103 tablet_service.cc:1467] Processing CreateTablet for tablet 41a78854de7a4d1287a4c000a3b20bd1 (DEFAULT_TABLE table=test-table [id=aaf13f94443d448b8f997ad3b0a8fd14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:27:50.863418 2904 tablet_service.cc:1467] Processing CreateTablet for tablet 41a78854de7a4d1287a4c000a3b20bd1 (DEFAULT_TABLE table=test-table [id=aaf13f94443d448b8f997ad3b0a8fd14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:27:50.863904 2695 tablet_service.cc:1467] Processing CreateTablet for tablet 41a78854de7a4d1287a4c000a3b20bd1 (DEFAULT_TABLE table=test-table [id=aaf13f94443d448b8f997ad3b0a8fd14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:27:50.864624 2904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41a78854de7a4d1287a4c000a3b20bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:50.864626 3103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41a78854de7a4d1287a4c000a3b20bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:50.864919 2695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41a78854de7a4d1287a4c000a3b20bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:50.871734 3219 tablet_bootstrap.cc:492] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769: Bootstrap starting.
I20260810 07:27:50.872576 3221 tablet_bootstrap.cc:492] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6: Bootstrap starting.
I20260810 07:27:50.873276 3220 tablet_bootstrap.cc:492] T 41a78854de7a4d1287a4c000a3b20bd1 P f4c5fa460e5a4a648bda699e263e0940: Bootstrap starting.
I20260810 07:27:50.874003 3219 tablet_bootstrap.cc:654] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:50.874773 3219 log.cc:824] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:50.874861 3221 tablet_bootstrap.cc:654] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:50.875892 3220 tablet_bootstrap.cc:654] T 41a78854de7a4d1287a4c000a3b20bd1 P f4c5fa460e5a4a648bda699e263e0940: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:50.876272 3221 log.cc:824] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:50.876732 3220 log.cc:824] T 41a78854de7a4d1287a4c000a3b20bd1 P f4c5fa460e5a4a648bda699e263e0940: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:50.876773 3219 tablet_bootstrap.cc:492] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769: No bootstrap required, opened a new log
I20260810 07:27:50.876988 3219 ts_tablet_manager.cc:1397] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260810 07:27:50.878033 3221 tablet_bootstrap.cc:492] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6: No bootstrap required, opened a new log
I20260810 07:27:50.878228 3221 ts_tablet_manager.cc:1397] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 07:27:50.880445 3219 raft_consensus.cc:348] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 } } peers { permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 } } peers { permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 } }
I20260810 07:27:50.880690 3219 raft_consensus.cc:374] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:50.880759 3219 raft_consensus.cc:729] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcf4b29ce0ac4dcc97cf34daed3fe769, State: Initialized, Role: FOLLOWER
I20260810 07:27:50.881090 3221 raft_consensus.cc:348] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 } } peers { permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 } } peers { permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 } }
I20260810 07:27:50.881323 3221 raft_consensus.cc:374] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:50.881268 3219 consensus_queue.cc:260] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 } } peers { permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 } } peers { permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 } }
I20260810 07:27:50.881394 3221 raft_consensus.cc:729] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ba31ad84ad84a7eb41bae121def8fc6, State: Initialized, Role: FOLLOWER
I20260810 07:27:50.881798 3221 consensus_queue.cc:260] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 } } peers { permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 } } peers { permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 } }
I20260810 07:27:50.882319 3219 ts_tablet_manager.cc:1428] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 07:27:50.882580 3196 heartbeater.cc:503] Master 127.12.157.254:39709 was elected leader, sending a full tablet report...
I20260810 07:27:50.882671 2809 heartbeater.cc:503] Master 127.12.157.254:39709 was elected leader, sending a full tablet report...
I20260810 07:27:50.882715 3221 ts_tablet_manager.cc:1428] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260810 07:27:50.883806 3220 tablet_bootstrap.cc:492] T 41a78854de7a4d1287a4c000a3b20bd1 P f4c5fa460e5a4a648bda699e263e0940: No bootstrap required, opened a new log
I20260810 07:27:50.884004 3220 ts_tablet_manager.cc:1397] T 41a78854de7a4d1287a4c000a3b20bd1 P f4c5fa460e5a4a648bda699e263e0940: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260810 07:27:50.887986 3220 raft_consensus.cc:348] T 41a78854de7a4d1287a4c000a3b20bd1 P f4c5fa460e5a4a648bda699e263e0940 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 } } peers { permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 } } peers { permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 } }
I20260810 07:27:50.888204 3220 raft_consensus.cc:374] T 41a78854de7a4d1287a4c000a3b20bd1 P f4c5fa460e5a4a648bda699e263e0940 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:50.888237 3220 raft_consensus.cc:729] T 41a78854de7a4d1287a4c000a3b20bd1 P f4c5fa460e5a4a648bda699e263e0940 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c5fa460e5a4a648bda699e263e0940, State: Initialized, Role: FOLLOWER
I20260810 07:27:50.888634 3220 consensus_queue.cc:260] T 41a78854de7a4d1287a4c000a3b20bd1 P f4c5fa460e5a4a648bda699e263e0940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 } } peers { permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 } } peers { permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 } }
I20260810 07:27:50.889329 3003 heartbeater.cc:503] Master 127.12.157.254:39709 was elected leader, sending a full tablet report...
I20260810 07:27:50.889626 3220 ts_tablet_manager.cc:1428] T 41a78854de7a4d1287a4c000a3b20bd1 P f4c5fa460e5a4a648bda699e263e0940: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260810 07:27:51.051949 3197 tablet.cc:2369] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 07:27:51.063405 3005 tablet.cc:2369] T 41a78854de7a4d1287a4c000a3b20bd1 P f4c5fa460e5a4a648bda699e263e0940: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 07:27:51.085758 2811 tablet.cc:2369] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:27:51.099327 3227 raft_consensus.cc:482] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:27:51.099617 3227 raft_consensus.cc:504] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 } } peers { permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 } } peers { permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 } }
I20260810 07:27:51.100914 3227 leader_election.cc:288] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4c5fa460e5a4a648bda699e263e0940 (127.12.157.194:36057), dcf4b29ce0ac4dcc97cf34daed3fe769 (127.12.157.193:41497)
I20260810 07:27:51.106650 3218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:51.101536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:51841 (local address 127.12.157.193:41497)
0810 07:27:51.101707 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:51.101713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:51.101741 (+ 28us) server_negotiation.cc:407] Connection header received
0810 07:27:51.101805 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:51.101811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:51.101866 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:51.101953 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:51.102380 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:51.103205 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:51.105098 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:51.105766 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:51.105878 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:51.105931 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:51.106402 (+ 471us) server_negotiation.cc:299] Negotiation successful
0810 07:27:51.106532 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 07:27:51.106650 3211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:51.101355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:53727 (local address 127.12.157.194:36057)
0810 07:27:51.101503 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:51.101507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:51.102057 (+ 550us) server_negotiation.cc:407] Connection header received
0810 07:27:51.102154 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:51.102156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:51.102201 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:51.102271 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:51.102554 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:51.103284 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:51.105093 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:51.105725 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:51.105874 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:51.105931 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:51.106413 (+ 482us) server_negotiation.cc:299] Negotiation successful
0810 07:27:51.106509 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 07:27:51.106699 3198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:51.101277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:41497 (local address 127.12.157.195:51841)
0810 07:27:51.101389 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:51.101406 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:51.101512 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:51.102012 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:51.102016 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:51.102061 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:51.102211 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:51.102217 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:51.103359 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:51.103365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:51.104962 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:51.104975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:51.105912 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:51.105917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:51.106174 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:51.106252 (+ 78us) client_negotiation.cc:769] Sending connection context
0810 07:27:51.106344 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 07:27:51.106509 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":282,"spinlock_wait_cycles":1920}
I20260810 07:27:51.106715 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:51.101284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:36057 (local address 127.12.157.195:53727)
0810 07:27:51.101916 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:51.101951 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:51.102068 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:51.102314 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:51.102319 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:51.102341 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:51.102443 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:51.102447 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:51.103393 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:51.103395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:51.104962 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:51.104975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:51.105912 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:51.105917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:51.106178 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:51.106252 (+ 74us) client_negotiation.cc:769] Sending connection context
0810 07:27:51.106419 (+ 167us) client_negotiation.cc:240] Negotiation successful
0810 07:27:51.106509 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":499,"spinlock_wait_cycles":30848,"thread_start_us":141,"threads_started":1}
I20260810 07:27:51.107468 2732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a78854de7a4d1287a4c000a3b20bd1" candidate_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" is_pre_election: true
I20260810 07:27:51.107664 2939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a78854de7a4d1287a4c000a3b20bd1" candidate_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c5fa460e5a4a648bda699e263e0940" is_pre_election: true
I20260810 07:27:51.107810 2732 raft_consensus.cc:2454] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ba31ad84ad84a7eb41bae121def8fc6 in term 0.
I20260810 07:27:51.108014 2939 raft_consensus.cc:2454] T 41a78854de7a4d1287a4c000a3b20bd1 P f4c5fa460e5a4a648bda699e263e0940 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ba31ad84ad84a7eb41bae121def8fc6 in term 0.
I20260810 07:27:51.108347 3037 leader_election.cc:302] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ba31ad84ad84a7eb41bae121def8fc6, dcf4b29ce0ac4dcc97cf34daed3fe769; no voters:
I20260810 07:27:51.108629 3227 raft_consensus.cc:2781] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:27:51.108721 3227 raft_consensus.cc:482] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:27:51.108767 3227 raft_consensus.cc:3037] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:51.109964 3227 raft_consensus.cc:504] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 } } peers { permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 } } peers { permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 } }
I20260810 07:27:51.110321 3227 leader_election.cc:288] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6 [CANDIDATE]: Term 1 election: Requested vote from peers f4c5fa460e5a4a648bda699e263e0940 (127.12.157.194:36057), dcf4b29ce0ac4dcc97cf34daed3fe769 (127.12.157.193:41497)
I20260810 07:27:51.110697 2732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a78854de7a4d1287a4c000a3b20bd1" candidate_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769"
I20260810 07:27:51.110740 2939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a78854de7a4d1287a4c000a3b20bd1" candidate_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c5fa460e5a4a648bda699e263e0940"
I20260810 07:27:51.110812 2732 raft_consensus.cc:3037] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:51.110915 2939 raft_consensus.cc:3037] T 41a78854de7a4d1287a4c000a3b20bd1 P f4c5fa460e5a4a648bda699e263e0940 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:51.111975 2732 raft_consensus.cc:2454] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ba31ad84ad84a7eb41bae121def8fc6 in term 1.
I20260810 07:27:51.112056 2939 raft_consensus.cc:2454] T 41a78854de7a4d1287a4c000a3b20bd1 P f4c5fa460e5a4a648bda699e263e0940 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ba31ad84ad84a7eb41bae121def8fc6 in term 1.
I20260810 07:27:51.112306 3037 leader_election.cc:302] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ba31ad84ad84a7eb41bae121def8fc6, dcf4b29ce0ac4dcc97cf34daed3fe769; no voters:
I20260810 07:27:51.112501 3227 raft_consensus.cc:2781] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:27:51.112776 3227 raft_consensus.cc:686] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6 [term 1 LEADER]: Becoming Leader. State: Replica: 5ba31ad84ad84a7eb41bae121def8fc6, State: Running, Role: LEADER
I20260810 07:27:51.113161 3227 consensus_queue.cc:237] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 } } peers { permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 } } peers { permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 } }
I20260810 07:27:51.116415 2492 catalog_manager.cc:5495] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ba31ad84ad84a7eb41bae121def8fc6 (127.12.157.195). New cstate: current_term: 1 leader_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 } health_report { overall_health: UNKNOWN } } }
I20260810 07:27:51.155473 3213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:51.150546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48052 (local address 127.12.157.195:42671)
0810 07:27:51.150699 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:51.150705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:51.150722 (+ 17us) server_negotiation.cc:407] Connection header received
0810 07:27:51.150854 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:51.150857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:51.150930 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:51.151053 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:27:51.151402 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:51.152283 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:51.153443 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:51.153813 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:51.153930 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:51.154392 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:27:51.155014 (+ 622us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:27:51.155137 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:51.155228 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 07:27:51.155303 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 07:27:51.162436 2732 raft_consensus.cc:1261] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 1 FOLLOWER]: Refusing update from remote peer 5ba31ad84ad84a7eb41bae121def8fc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:27:51.162436 2939 raft_consensus.cc:1261] T 41a78854de7a4d1287a4c000a3b20bd1 P f4c5fa460e5a4a648bda699e263e0940 [term 1 FOLLOWER]: Refusing update from remote peer 5ba31ad84ad84a7eb41bae121def8fc6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:27:51.163237 3227 consensus_queue.cc:1035] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:27:51.163488 3234 consensus_queue.cc:1035] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:27:51.169596 3237 mvcc.cc:205] Tried to move back new op lower bound from 7316876784272916480 to 7316876784082153472. Current Snapshot: MvccSnapshot[applied={T|T < 7316876784272916480}]
I20260810 07:27:51.171391 3239 mvcc.cc:205] Tried to move back new op lower bound from 7316876784272916480 to 7316876784082153472. Current Snapshot: MvccSnapshot[applied={T|T < 7316876784272916480}]
I20260810 07:27:51.171540 3238 mvcc.cc:205] Tried to move back new op lower bound from 7316876784272916480 to 7316876784082153472. Current Snapshot: MvccSnapshot[applied={T|T < 7316876784272916480}]
I20260810 07:27:51.180857 2494 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:27:51.181093 2494 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:27:51.183794 2494 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a: Sending DeleteTablet for 3 replicas of tablet 41a78854de7a4d1287a4c000a3b20bd1
I20260810 07:27:51.184664 2904 tablet_service.cc:1514] Processing DeleteTablet for tablet 41a78854de7a4d1287a4c000a3b20bd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:27:51 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:39678
I20260810 07:27:51.184700 3103 tablet_service.cc:1514] Processing DeleteTablet for tablet 41a78854de7a4d1287a4c000a3b20bd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:27:51 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:48044
I20260810 07:27:51.185231 3255 tablet_replica.cc:330] T 41a78854de7a4d1287a4c000a3b20bd1 P f4c5fa460e5a4a648bda699e263e0940: stopping tablet replica
I20260810 07:27:51.185227 3253 tablet_replica.cc:330] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6: stopping tablet replica
I20260810 07:27:51.185266 2695 tablet_service.cc:1514] Processing DeleteTablet for tablet 41a78854de7a4d1287a4c000a3b20bd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:27:51 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:37782
I20260810 07:27:51.185587 3255 raft_consensus.cc:2229] T 41a78854de7a4d1287a4c000a3b20bd1 P f4c5fa460e5a4a648bda699e263e0940 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:27:51.185701 3253 raft_consensus.cc:2229] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:27:51.185931 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 2588
I20260810 07:27:51.185937 3255 raft_consensus.cc:2258] T 41a78854de7a4d1287a4c000a3b20bd1 P f4c5fa460e5a4a648bda699e263e0940 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:27:51.186249 3253 raft_consensus.cc:2258] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:27:51.196081 3255 ts_tablet_manager.cc:1905] T 41a78854de7a4d1287a4c000a3b20bd1 P f4c5fa460e5a4a648bda699e263e0940: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:27:51.200064 3255 ts_tablet_manager.cc:1918] T 41a78854de7a4d1287a4c000a3b20bd1 P f4c5fa460e5a4a648bda699e263e0940: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:27:51.200207 3255 log.cc:1196] T 41a78854de7a4d1287a4c000a3b20bd1 P f4c5fa460e5a4a648bda699e263e0940: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/wal/wals/41a78854de7a4d1287a4c000a3b20bd1
I20260810 07:27:51.200567 3255 ts_tablet_manager.cc:1939] T 41a78854de7a4d1287a4c000a3b20bd1 P f4c5fa460e5a4a648bda699e263e0940: Deleting consensus metadata
I20260810 07:27:51.201612 3253 ts_tablet_manager.cc:1905] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:27:51.202164 2471 catalog_manager.cc:4841] TS f4c5fa460e5a4a648bda699e263e0940 (127.12.157.194:36057): tablet 41a78854de7a4d1287a4c000a3b20bd1 (table test-table [id=aaf13f94443d448b8f997ad3b0a8fd14]) successfully deleted
W20260810 07:27:51.202284 2473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.157.193:41497 (error 108)
I20260810 07:27:51.203184 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 2816
I20260810 07:27:51.206249 3212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:51.203214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:41497 (local address 127.0.0.1:37792)
0810 07:27:51.206049 (+ 2835us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:51.206138 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.193:41497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2717}
W20260810 07:27:51.212204 2473 catalog_manager.cc:4584] TS dcf4b29ce0ac4dcc97cf34daed3fe769 (127.12.157.193:41497): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 41a78854de7a4d1287a4c000a3b20bd1: Network error: Client connection negotiation failed: client connection to 127.12.157.193:41497: connect: Connection refused (error 111)
I20260810 07:27:51.218717 3253 ts_tablet_manager.cc:1918] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:27:51.218886 3253 log.cc:1196] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/wal/wals/41a78854de7a4d1287a4c000a3b20bd1
I20260810 07:27:51.219244 3253 ts_tablet_manager.cc:1939] T 41a78854de7a4d1287a4c000a3b20bd1 P 5ba31ad84ad84a7eb41bae121def8fc6: Deleting consensus metadata
I20260810 07:27:51.220817 2473 catalog_manager.cc:4841] TS 5ba31ad84ad84a7eb41bae121def8fc6 (127.12.157.195:42671): tablet 41a78854de7a4d1287a4c000a3b20bd1 (table test-table [id=aaf13f94443d448b8f997ad3b0a8fd14]) successfully deleted
I20260810 07:27:51.222906 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 3010
I20260810 07:27:51.243392 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 2440
I20260810 07:27:51.256844 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:39709
--webserver_interface=127.12.157.254
--webserver_port=41799
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:46101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:39709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:51.371127 3258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:51.371392 3258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:51.371433 3258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:51.375631 3258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:27:51.375694 3258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:51.375712 3258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:51.375730 3258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:27:51.375746 3258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:27:51.381070 3258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:46101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:39709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:39709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=41799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:27:51.382107 3258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:51.383102 3258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:51.396876 3267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:51.396878 3268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:51.396876 3276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:51.397153 3258 server_base.cc:1034] running on GCE node
I20260810 07:27:51.397836 3258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:51.399017 3258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:51.400221 3258 hybrid_clock.cc:648] HybridClock initialized: now 1786346871400205 us; error 26 us; skew 500 ppm
Aug 10 07:27:51 dist-test-slave-ffrd krb5kdc[2426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346871, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:51.403158 3258 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:27:51.404129 3258 webserver.cc:460] Webserver started at http://127.12.157.254:41799/ using document root <none> and password file <none>
I20260810 07:27:51.404685 3258 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:51.404760 3258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:51.408613 3258 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:27:51.410881 3288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:51.412117 3258 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 07:27:51.412230 3258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "c72a6929623c409db8ae2e39c8aacf9a"
format_stamp: "Formatted at 2026-08-10 07:27:49 on dist-test-slave-ffrd"
server_key: "bf0dd917acaa08a98ffde4343e253ad4"
server_key_iv: "95f337f7d79b3b8a200c523842766a67"
server_key_version: "encryptionkey@0"
I20260810 07:27:51.412822 3258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:51.428560 3258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:51.431864 3258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:51.432034 3258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:51.440033 3258 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:39709
I20260810 07:27:51.440327 3374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:39709 every 8 connection(s)
I20260810 07:27:51.441241 3258 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:27:51.444350 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 3258
I20260810 07:27:51.444836 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:41497
--local_ip_for_outbound_sockets=127.12.157.193
--tserver_master_addrs=127.12.157.254:39709
--webserver_port=46173
--webserver_interface=127.12.157.193
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:46101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:27:51.446172 3375 sys_catalog.cc:263] Verifying existing consensus state
I20260810 07:27:51.448248 3375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a: Bootstrap starting.
I20260810 07:27:51.468102 3375 log.cc:824] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:51.477219 3375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 07:27:51.477646 3375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a: Bootstrap complete.
I20260810 07:27:51.481946 3375 raft_consensus.cc:348] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72a6929623c409db8ae2e39c8aacf9a" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39709 } }
I20260810 07:27:51.482350 3375 raft_consensus.cc:729] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c72a6929623c409db8ae2e39c8aacf9a, State: Initialized, Role: FOLLOWER
I20260810 07:27:51.482947 3375 consensus_queue.cc:260] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72a6929623c409db8ae2e39c8aacf9a" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39709 } }
I20260810 07:27:51.483117 3375 raft_consensus.cc:388] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:27:51.483201 3375 raft_consensus.cc:482] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:27:51.483305 3375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [term 1 FOLLOWER]: Advancing to term 2
I20260810 07:27:51.486358 3375 raft_consensus.cc:504] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72a6929623c409db8ae2e39c8aacf9a" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39709 } }
I20260810 07:27:51.486799 3375 leader_election.cc:302] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c72a6929623c409db8ae2e39c8aacf9a; no voters:
I20260810 07:27:51.487200 3375 leader_election.cc:288] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 07:27:51.487802 3392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [term 2 FOLLOWER]: Leader election won for term 2
I20260810 07:27:51.488629 3375 sys_catalog.cc:564] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:27:51.488637 3392 raft_consensus.cc:686] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [term 2 LEADER]: Becoming Leader. State: Replica: c72a6929623c409db8ae2e39c8aacf9a, State: Running, Role: LEADER
I20260810 07:27:51.489174 3392 consensus_queue.cc:237] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72a6929623c409db8ae2e39c8aacf9a" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39709 } }
I20260810 07:27:51.490825 3394 sys_catalog.cc:455] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c72a6929623c409db8ae2e39c8aacf9a. Latest consensus state: current_term: 2 leader_uuid: "c72a6929623c409db8ae2e39c8aacf9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72a6929623c409db8ae2e39c8aacf9a" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39709 } } }
I20260810 07:27:51.491238 3388 sys_catalog.cc:455] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c72a6929623c409db8ae2e39c8aacf9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72a6929623c409db8ae2e39c8aacf9a" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39709 } } }
I20260810 07:27:51.491638 3394 sys_catalog.cc:458] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:51.491640 3388 sys_catalog.cc:458] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:51.494567 3410 catalog_manager.cc:1259] Loaded cluster ID: d721cb79fc0641e0b89ce9aae9674901
I20260810 07:27:51.494632 3410 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a: loading cluster ID for follower catalog manager: success
W20260810 07:27:51.495667 3410 catalog_manager.cc:1576] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260810 07:27:51.495746 3410 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260810 07:27:51.495955 3411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:27:51.498602 3411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:27:51.499101 3411 catalog_manager.cc:1259] Loaded cluster ID: d721cb79fc0641e0b89ce9aae9674901
I20260810 07:27:51.499166 3411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:27:51.500244 3411 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 07:27:51.505205 3411 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 07:27:51.511622 3411 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:27:51.512559 3411 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a: Loaded TSK: 0
I20260810 07:27:51.513957 3411 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 07:27:51.572551 3384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:51.572866 3384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:51.572996 3384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:51.577136 3384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:51.577224 3384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:51.577313 3384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:27:51.582723 3384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:46101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:41497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=46173
--tserver_master_addrs=127.12.157.254:39709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3384
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:27:51.583833 3384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:51.584887 3384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:51.594997 3422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:51.595011 3423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:51.595227 3384 server_base.cc:1034] running on GCE node
W20260810 07:27:51.595301 3426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:51.595768 3384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:51.596455 3384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:51.597641 3384 hybrid_clock.cc:648] HybridClock initialized: now 1786346871597617 us; error 42 us; skew 500 ppm
Aug 10 07:27:51 dist-test-slave-ffrd krb5kdc[2426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346871, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:51.600448 3384 init.cc:377] Logged in from keytab as kudu/127.12.157.193@KRBTEST.COM (short username kudu)
I20260810 07:27:51.601442 3384 webserver.cc:460] Webserver started at http://127.12.157.193:46173/ using document root <none> and password file <none>
I20260810 07:27:51.602010 3384 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:51.602087 3384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:51.606528 3384 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 07:27:51.608827 3440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:51.610183 3384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:27:51.610338 3384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769"
format_stamp: "Formatted at 2026-08-10 07:27:50 on dist-test-slave-ffrd"
server_key: "b6b72b66603fd0e3ccedf473e1313806"
server_key_iv: "58712e49785086b3d04a96752a52aca4"
server_key_version: "encryptionkey@0"
I20260810 07:27:51.610886 3384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:51.654510 3384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:51.657670 3384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:51.657891 3384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:51.658543 3384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:51.659713 3456 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 07:27:51.664215 3384 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 07:27:51.664316 3384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260810 07:27:51.664441 3384 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 07:27:51.666332 3384 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 07:27:51.666500 3384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260810 07:27:51.666790 3456 tablet_bootstrap.cc:492] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769: Bootstrap starting.
I20260810 07:27:51.678227 3384 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:41497
I20260810 07:27:51.678315 3636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:41497 every 8 connection(s)
I20260810 07:27:51.679260 3384 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:27:51.684058 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 3384
I20260810 07:27:51.684690 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:36057
--local_ip_for_outbound_sockets=127.12.157.194
--tserver_master_addrs=127.12.157.254:39709
--webserver_port=35389
--webserver_interface=127.12.157.194
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:46101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:27:51.687124 3456 log.cc:824] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769: Log is configured to *not* fsync() on all Append() calls
Aug 10 07:27:51 dist-test-slave-ffrd krb5kdc[2426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346871, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:27:51.695850 3456 tablet_bootstrap.cc:492] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 07:27:51.696345 3456 tablet_bootstrap.cc:492] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769: Bootstrap complete.
I20260810 07:27:51.697933 3456 ts_tablet_manager.cc:1397] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769: Time spent bootstrapping tablet: real 0.031s user 0.014s sys 0.015s
I20260810 07:27:51.701454 3644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:51.681359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.193:57119)
0810 07:27:51.682229 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:51.682287 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:51.683590 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:51.685482 (+ 1892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:51.685490 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:51.685970 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:51.686443 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:51.686458 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:51.688485 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:51.688488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:51.689877 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:51.689900 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:51.691117 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:51.691121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:51.691301 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:51.692047 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:51.692067 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:51.694266 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:51.697191 (+ 2925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:51.697199 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:51.697217 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:51.697633 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:51.698117 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:51.698121 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:51.698122 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:51.698291 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:51.699037 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:51.699044 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:51.699322 (+ 278us) client_negotiation.cc:769] Sending connection context
0810 07:27:51.699660 (+ 338us) client_negotiation.cc:240] Negotiation successful
0810 07:27:51.700027 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":162,"threads_started":1}
I20260810 07:27:51.701606 3456 raft_consensus.cc:348] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 } } peers { permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 } } peers { permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 } }
I20260810 07:27:51.702163 3456 raft_consensus.cc:729] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcf4b29ce0ac4dcc97cf34daed3fe769, State: Initialized, Role: FOLLOWER
I20260810 07:27:51.702684 3456 consensus_queue.cc:260] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 } } peers { permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 } } peers { permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 } }
I20260810 07:27:51.703207 3641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:51.681442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:57119 (local address 127.12.157.254:39709)
0810 07:27:51.681877 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:51.681909 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:51.682455 (+ 546us) server_negotiation.cc:407] Connection header received
0810 07:27:51.684629 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:51.684640 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:51.685078 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:51.685435 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:51.686684 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:51.688307 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:51.690040 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:51.690875 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:51.691179 (+ 304us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:51.694534 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:51.694541 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:51.694557 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:51.694593 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:51.697045 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:51.697781 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:51.697791 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:51.697808 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:51.697989 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:51.698407 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:51.698411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:51.698412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:51.698887 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:51.701879 (+ 2992us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:51.702047 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 07:27:51.702396 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":150,"threads_started":1}
I20260810 07:27:51.703773 3456 ts_tablet_manager.cc:1428] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260810 07:27:51.706151 3637 heartbeater.cc:347] Connected to a master server at 127.12.157.254:39709
I20260810 07:27:51.706491 3637 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:51.707048 3637 heartbeater.cc:511] Master 127.12.157.254:39709 requested a full tablet report, sending...
I20260810 07:27:51.709977 3321 ts_manager.cc:194] Registered new tserver with Master: dcf4b29ce0ac4dcc97cf34daed3fe769 (127.12.157.193:41497)
I20260810 07:27:51.714246 3321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.193@KRBTEST.COM'} at 127.12.157.193:57119
I20260810 07:27:51.717144 3637 heartbeater.cc:503] Master 127.12.157.254:39709 was elected leader, sending a full tablet report...
Aug 10 07:27:51 dist-test-slave-ffrd krb5kdc[2426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346871, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for kudu/127.12.157.193@KRBTEST.COM
I20260810 07:27:51.725375 3652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:51.713727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37808 (local address 127.12.157.193:41497)
0810 07:27:51.714150 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:51.714163 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:51.714604 (+ 441us) server_negotiation.cc:407] Connection header received
0810 07:27:51.714911 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:51.714922 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:51.715136 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:51.715276 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:51.716177 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:51.717313 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:51.718421 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:51.718836 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:51.718975 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:51.721741 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:51.721750 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:51.721760 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:51.721784 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:51.723441 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:51.723993 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:51.723998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:51.723999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:51.724078 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:51.724366 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:51.724368 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:51.724370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:51.724597 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:51.724701 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:51.725103 (+ 402us) server_negotiation.cc:299] Negotiation successful
0810 07:27:51.725246 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":92,"threads_started":1}
I20260810 07:27:51.725485 3651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:51.713634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:41497 (local address 127.0.0.1:37808)
0810 07:27:51.714424 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:51.714475 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:51.714750 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:51.715287 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:51.715292 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:51.715900 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:51.716039 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:51.716048 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:51.717537 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:51.717541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:51.718288 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:51.718299 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:51.718978 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:51.718983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:51.719103 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:51.719673 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:51.719688 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:51.721595 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:51.723560 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:51.723570 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:51.723575 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:51.723872 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:51.724168 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:51.724170 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:51.724172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:51.724272 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:51.724710 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:51.724715 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:51.724851 (+ 136us) client_negotiation.cc:769] Sending connection context
0810 07:27:51.725031 (+ 180us) client_negotiation.cc:240] Negotiation successful
0810 07:27:51.725263 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":134,"threads_started":1}
I20260810 07:27:51.726555 3538 tablet_service.cc:1514] Processing DeleteTablet for tablet 41a78854de7a4d1287a4c000a3b20bd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:27:51 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:37808
I20260810 07:27:51.727622 3653 tablet_replica.cc:330] stopping tablet replica
I20260810 07:27:51.727970 3653 raft_consensus.cc:2229] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:27:51.728240 3653 raft_consensus.cc:2258] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:27:51.729348 3653 ts_tablet_manager.cc:1905] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:27:51.733352 3653 ts_tablet_manager.cc:1918] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:27:51.733471 3653 log.cc:1196] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/wal/wals/41a78854de7a4d1287a4c000a3b20bd1
I20260810 07:27:51.733827 3653 ts_tablet_manager.cc:1939] T 41a78854de7a4d1287a4c000a3b20bd1 P dcf4b29ce0ac4dcc97cf34daed3fe769: Deleting consensus metadata
I20260810 07:27:51.735603 3295 catalog_manager.cc:4841] TS dcf4b29ce0ac4dcc97cf34daed3fe769 (127.12.157.193:41497): tablet 41a78854de7a4d1287a4c000a3b20bd1 (table test-table [id=aaf13f94443d448b8f997ad3b0a8fd14]) successfully deleted
W20260810 07:27:51.812889 3646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:51.813175 3646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:51.813221 3646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:51.817543 3646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:51.817610 3646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:51.817672 3646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:27:51.822896 3646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:46101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:36057
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=35389
--tserver_master_addrs=127.12.157.254:39709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:27:51.823927 3646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:51.824934 3646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:51.834951 3667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:51.835031 3664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:51.834951 3662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:51.835259 3646 server_base.cc:1034] running on GCE node
I20260810 07:27:51.835618 3646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:51.836302 3646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:51.837466 3646 hybrid_clock.cc:648] HybridClock initialized: now 1786346871837451 us; error 25 us; skew 500 ppm
Aug 10 07:27:51 dist-test-slave-ffrd krb5kdc[2426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346871, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:51.840235 3646 init.cc:377] Logged in from keytab as kudu/127.12.157.194@KRBTEST.COM (short username kudu)
I20260810 07:27:51.841156 3646 webserver.cc:460] Webserver started at http://127.12.157.194:35389/ using document root <none> and password file <none>
I20260810 07:27:51.841677 3646 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:51.841750 3646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:51.845508 3646 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260810 07:27:51.847581 3680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:51.848819 3646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 07:27:51.848999 3646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "f4c5fa460e5a4a648bda699e263e0940"
format_stamp: "Formatted at 2026-08-10 07:27:50 on dist-test-slave-ffrd"
server_key: "c0397f75dfbe44945d29906535c8fb88"
server_key_iv: "31ca7d568640b81076c0f66897d7ffbc"
server_key_version: "encryptionkey@0"
I20260810 07:27:51.849557 3646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:51.864661 3646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:51.867919 3646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:51.868147 3646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:51.868782 3646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:51.869772 3646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:51.869846 3646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:51.869953 3646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:51.870007 3646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:51.879489 3646 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:36057
I20260810 07:27:51.879520 3859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:36057 every 8 connection(s)
I20260810 07:27:51.880493 3646 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:27:51.883584 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 3646
I20260810 07:27:51.884161 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:42671
--local_ip_for_outbound_sockets=127.12.157.195
--tserver_master_addrs=127.12.157.254:39709
--webserver_port=46713
--webserver_interface=127.12.157.195
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:46101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 07:27:51 dist-test-slave-ffrd krb5kdc[2426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346871, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:27:51.895619 3641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:51.882344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:50701 (local address 127.12.157.254:39709)
0810 07:27:51.882501 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:51.882505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:51.883321 (+ 816us) server_negotiation.cc:407] Connection header received
0810 07:27:51.885445 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:51.885449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:51.885501 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:51.885575 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:51.887021 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:51.887565 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:51.888790 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:51.889111 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:51.889230 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:51.892021 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:51.892025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:51.892027 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:51.892053 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:51.893548 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:51.894063 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:51.894068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:51.894070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:51.894162 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:51.894514 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:51.894518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:51.894520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:51.894706 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:51.894820 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:51.895305 (+ 485us) server_negotiation.cc:299] Negotiation successful
0810 07:27:51.895444 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 07:27:51.896417 3864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:51.882706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.194:50701)
0810 07:27:51.883146 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:51.883188 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:51.885202 (+ 2014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:51.885792 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:51.885804 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:51.886350 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:51.886861 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:51.886877 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:51.887702 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:51.887706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:51.888659 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:51.888669 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:51.889275 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:51.889280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:51.889545 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:51.890247 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:51.890264 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:51.891832 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:51.893671 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:51.893677 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:51.893692 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:51.893956 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:51.894263 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:51.894266 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:51.894268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:51.894413 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:51.894819 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:51.894825 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:51.895072 (+ 247us) client_negotiation.cc:769] Sending connection context
0810 07:27:51.895262 (+ 190us) client_negotiation.cc:240] Negotiation successful
0810 07:27:51.895599 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":111,"threads_started":1}
I20260810 07:27:51.897661 3861 heartbeater.cc:347] Connected to a master server at 127.12.157.254:39709
I20260810 07:27:51.897922 3861 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:51.898422 3861 heartbeater.cc:511] Master 127.12.157.254:39709 requested a full tablet report, sending...
I20260810 07:27:51.899490 3321 ts_manager.cc:194] Registered new tserver with Master: f4c5fa460e5a4a648bda699e263e0940 (127.12.157.194:36057)
I20260810 07:27:51.900056 3321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.194@KRBTEST.COM'} at 127.12.157.194:50701
W20260810 07:27:52.002962 3866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:52.003309 3866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:52.003409 3866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:52.007740 3866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:52.007846 3866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:52.007937 3866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:27:52.013798 3866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:46101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:42671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=46713
--tserver_master_addrs=127.12.157.254:39709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:27:52.015144 3866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:52.016330 3866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:52.026569 3876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:52.026557 3879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:52.026564 3877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:52.026801 3866 server_base.cc:1034] running on GCE node
I20260810 07:27:52.027372 3866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:52.028047 3866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:52.029277 3866 hybrid_clock.cc:648] HybridClock initialized: now 1786346872029250 us; error 39 us; skew 500 ppm
Aug 10 07:27:52 dist-test-slave-ffrd krb5kdc[2426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346872, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:52.032505 3866 init.cc:377] Logged in from keytab as kudu/127.12.157.195@KRBTEST.COM (short username kudu)
I20260810 07:27:52.033555 3866 webserver.cc:460] Webserver started at http://127.12.157.195:46713/ using document root <none> and password file <none>
I20260810 07:27:52.034159 3866 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:52.034282 3866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:52.038401 3866 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 07:27:52.040663 3894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:52.042273 3866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:27:52.042439 3866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "5ba31ad84ad84a7eb41bae121def8fc6"
format_stamp: "Formatted at 2026-08-10 07:27:50 on dist-test-slave-ffrd"
server_key: "8d14b162bd5a5b6a15b81be775336175"
server_key_iv: "72e1b38a26936d3b2ec5a90c620d2883"
server_key_version: "encryptionkey@0"
I20260810 07:27:52.043056 3866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:52.077369 3866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:52.080554 3866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:52.080785 3866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:52.081502 3866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:52.082423 3866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:52.082495 3866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:52.082567 3866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:52.082620 3866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:52.092202 3866 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:42671
I20260810 07:27:52.092290 4089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:42671 every 8 connection(s)
I20260810 07:27:52.093534 3866 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:27:52.102777 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 3866
Aug 10 07:27:52 dist-test-slave-ffrd krb5kdc[2426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346872, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:27:52.107628 3641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:52.095417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:51621 (local address 127.12.157.254:39709)
0810 07:27:52.095573 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:52.095576 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:52.096399 (+ 823us) server_negotiation.cc:407] Connection header received
0810 07:27:52.097914 (+ 1515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:52.097917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:52.097979 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:52.098081 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:52.099238 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:52.099765 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:52.100903 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:52.101226 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:52.101340 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:52.104020 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:52.104023 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:52.104025 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:52.104049 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:52.105551 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:52.106103 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:52.106107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:52.106109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:52.106194 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:52.106548 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:52.106552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:52.106554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:52.106730 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:52.106862 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:52.107334 (+ 472us) server_negotiation.cc:299] Negotiation successful
0810 07:27:52.107456 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 07:27:52.108642 4093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:52.095786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.195:51621)
0810 07:27:52.096252 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:52.096294 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:52.097689 (+ 1395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:52.098265 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:52.098272 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:52.098623 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:52.099087 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:52.099100 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:52.099903 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:52.099906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:52.100763 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:52.100772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:52.101384 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:52.101388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:52.101557 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:52.102124 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:52.102140 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:52.103848 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:52.105674 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:52.105680 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:52.105693 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:52.105995 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:52.106298 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:52.106301 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:52.106303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:52.106447 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:52.106842 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:52.106849 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:52.107098 (+ 249us) client_negotiation.cc:769] Sending connection context
0810 07:27:52.107286 (+ 188us) client_negotiation.cc:240] Negotiation successful
0810 07:27:52.107598 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":120,"threads_started":1}
I20260810 07:27:52.110005 4091 heartbeater.cc:347] Connected to a master server at 127.12.157.254:39709
I20260810 07:27:52.110281 4091 heartbeater.cc:464] Registering TS with master...
I20260810 07:27:52.110891 4091 heartbeater.cc:511] Master 127.12.157.254:39709 requested a full tablet report, sending...
I20260810 07:27:52.112069 3321 ts_manager.cc:194] Registered new tserver with Master: 5ba31ad84ad84a7eb41bae121def8fc6 (127.12.157.195:42671)
I20260810 07:27:52.112636 3321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.195@KRBTEST.COM'} at 127.12.157.195:51621
I20260810 07:27:52.113592 4095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:52.103495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47246 (local address 127.12.157.254:39709)
0810 07:27:52.103727 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:52.103732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:52.103912 (+ 180us) server_negotiation.cc:407] Connection header received
0810 07:27:52.104069 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:52.104072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:52.104123 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:52.104223 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:52.105248 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:52.105772 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:52.106624 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:52.106855 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:52.106978 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:52.108579 (+ 1601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:52.108585 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:52.108590 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:52.108616 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:52.110641 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:52.111134 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:52.111137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:52.111139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:52.111228 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:52.111550 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:52.111552 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:52.111553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:52.111782 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:52.111978 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:52.113286 (+ 1308us) server_negotiation.cc:299] Negotiation successful
0810 07:27:52.113438 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":81,"threads_started":1}
I20260810 07:27:52.116355 12919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:27:52.120587 4095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:52.116919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47258 (local address 127.12.157.254:39709)
0810 07:27:52.117085 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:52.117089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:52.117232 (+ 143us) server_negotiation.cc:407] Connection header received
0810 07:27:52.117400 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:52.117403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:52.117498 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:52.117607 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:27:52.117849 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:52.118337 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:52.119181 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:52.119423 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:52.119539 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:52.119865 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:27:52.120211 (+ 346us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:27:52.120343 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:52.120409 (+ 66us) server_negotiation.cc:299] Negotiation successful
0810 07:27:52.120465 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 07:27:52.121282 3321 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:47258:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:27:52.124403 3321 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:27:52.132886 3538 tablet_service.cc:1467] Processing CreateTablet for tablet 927050ac821f4613b4ddb78d9ffb04b0 (DEFAULT_TABLE table=test-table [id=0d4a3861fc8645ed8ad9286715eed95e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:27:52.133633 3538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 927050ac821f4613b4ddb78d9ffb04b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:52.136714 3456 tablet_bootstrap.cc:492] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769: Bootstrap starting.
I20260810 07:27:52.137655 3456 tablet_bootstrap.cc:654] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:52.137769 3651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:52.131549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:42671 (local address 127.0.0.1:48064)
0810 07:27:52.131740 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:52.131758 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:52.131844 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:52.132884 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:52.132887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:52.132905 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:52.133069 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:52.133075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:52.134642 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:52.134646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:52.135851 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:52.135863 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:52.137269 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:52.137273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:52.137369 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:52.137420 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 07:27:52.137513 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 07:27:52.137622 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260810 07:27:52.138062 4110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:52.131844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:36057 (local address 127.0.0.1:39682)
0810 07:27:52.132170 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:52.132184 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:52.132263 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:52.134042 (+ 1779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:52.134045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:52.134138 (+ 93us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:52.134260 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:52.134268 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:52.135349 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:52.135354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:52.136216 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:52.136227 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:52.137498 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:52.137502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:52.137593 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:52.137623 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 07:27:52.137695 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 07:27:52.137767 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":1,"thread_start_us":118,"threads_started":1}
I20260810 07:27:52.138104 4108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:52.131767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48064 (local address 127.12.157.195:42671)
0810 07:27:52.132323 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:52.132336 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:52.132362 (+ 26us) server_negotiation.cc:407] Connection header received
0810 07:27:52.132454 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:52.132460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:52.132681 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:52.132871 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:52.133428 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:52.134499 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:52.136003 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:52.137129 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:52.137279 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:52.137428 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:52.137546 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 07:27:52.137646 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":199,"threads_started":1}
I20260810 07:27:52.138289 4111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:52.132114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39682 (local address 127.12.157.194:36057)
0810 07:27:52.133391 (+ 1277us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:52.133409 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:52.133434 (+ 25us) server_negotiation.cc:407] Connection header received
0810 07:27:52.133517 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:52.133524 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:52.133710 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:52.133856 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:52.134408 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:52.135215 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:52.136359 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:52.137193 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:52.137300 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:52.137427 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:52.138002 (+ 575us) server_negotiation.cc:299] Negotiation successful
0810 07:27:52.138109 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1021,"thread_start_us":122,"threads_started":1}
I20260810 07:27:52.139240 3456 tablet_bootstrap.cc:492] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769: No bootstrap required, opened a new log
I20260810 07:27:52.139322 3456 ts_tablet_manager.cc:1397] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 07:27:52.139902 3456 raft_consensus.cc:348] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 } } peers { permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 } } peers { permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 } }
I20260810 07:27:52.140033 3456 raft_consensus.cc:374] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:52.140079 3456 raft_consensus.cc:729] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcf4b29ce0ac4dcc97cf34daed3fe769, State: Initialized, Role: FOLLOWER
I20260810 07:27:52.140228 3456 consensus_queue.cc:260] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 } } peers { permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 } } peers { permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 } }
I20260810 07:27:52.140524 3456 ts_tablet_manager.cc:1428] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 07:27:52.141589 3755 tablet_service.cc:1467] Processing CreateTablet for tablet 927050ac821f4613b4ddb78d9ffb04b0 (DEFAULT_TABLE table=test-table [id=0d4a3861fc8645ed8ad9286715eed95e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:27:52.142606 3755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 927050ac821f4613b4ddb78d9ffb04b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:52.142637 3981 tablet_service.cc:1467] Processing CreateTablet for tablet 927050ac821f4613b4ddb78d9ffb04b0 (DEFAULT_TABLE table=test-table [id=0d4a3861fc8645ed8ad9286715eed95e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:27:52.143536 3981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 927050ac821f4613b4ddb78d9ffb04b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:27:52.150084 4115 tablet_bootstrap.cc:492] T 927050ac821f4613b4ddb78d9ffb04b0 P f4c5fa460e5a4a648bda699e263e0940: Bootstrap starting.
I20260810 07:27:52.150713 4117 tablet_bootstrap.cc:492] T 927050ac821f4613b4ddb78d9ffb04b0 P 5ba31ad84ad84a7eb41bae121def8fc6: Bootstrap starting.
I20260810 07:27:52.152777 4117 tablet_bootstrap.cc:654] T 927050ac821f4613b4ddb78d9ffb04b0 P 5ba31ad84ad84a7eb41bae121def8fc6: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:52.153326 4115 tablet_bootstrap.cc:654] T 927050ac821f4613b4ddb78d9ffb04b0 P f4c5fa460e5a4a648bda699e263e0940: Neither blocks nor log segments found. Creating new log.
I20260810 07:27:52.154237 4117 log.cc:824] T 927050ac821f4613b4ddb78d9ffb04b0 P 5ba31ad84ad84a7eb41bae121def8fc6: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:52.154238 4115 log.cc:824] T 927050ac821f4613b4ddb78d9ffb04b0 P f4c5fa460e5a4a648bda699e263e0940: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:52.156241 4115 tablet_bootstrap.cc:492] T 927050ac821f4613b4ddb78d9ffb04b0 P f4c5fa460e5a4a648bda699e263e0940: No bootstrap required, opened a new log
I20260810 07:27:52.156260 4117 tablet_bootstrap.cc:492] T 927050ac821f4613b4ddb78d9ffb04b0 P 5ba31ad84ad84a7eb41bae121def8fc6: No bootstrap required, opened a new log
I20260810 07:27:52.156448 4115 ts_tablet_manager.cc:1397] T 927050ac821f4613b4ddb78d9ffb04b0 P f4c5fa460e5a4a648bda699e263e0940: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260810 07:27:52.156575 4117 ts_tablet_manager.cc:1397] T 927050ac821f4613b4ddb78d9ffb04b0 P 5ba31ad84ad84a7eb41bae121def8fc6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 07:27:52.159431 4117 raft_consensus.cc:348] T 927050ac821f4613b4ddb78d9ffb04b0 P 5ba31ad84ad84a7eb41bae121def8fc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 } } peers { permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 } } peers { permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 } }
I20260810 07:27:52.159431 4115 raft_consensus.cc:348] T 927050ac821f4613b4ddb78d9ffb04b0 P f4c5fa460e5a4a648bda699e263e0940 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 } } peers { permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 } } peers { permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 } }
I20260810 07:27:52.159679 4117 raft_consensus.cc:374] T 927050ac821f4613b4ddb78d9ffb04b0 P 5ba31ad84ad84a7eb41bae121def8fc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:52.159673 4115 raft_consensus.cc:374] T 927050ac821f4613b4ddb78d9ffb04b0 P f4c5fa460e5a4a648bda699e263e0940 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:27:52.159763 4115 raft_consensus.cc:729] T 927050ac821f4613b4ddb78d9ffb04b0 P f4c5fa460e5a4a648bda699e263e0940 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c5fa460e5a4a648bda699e263e0940, State: Initialized, Role: FOLLOWER
I20260810 07:27:52.159763 4117 raft_consensus.cc:729] T 927050ac821f4613b4ddb78d9ffb04b0 P 5ba31ad84ad84a7eb41bae121def8fc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ba31ad84ad84a7eb41bae121def8fc6, State: Initialized, Role: FOLLOWER
I20260810 07:27:52.160240 4115 consensus_queue.cc:260] T 927050ac821f4613b4ddb78d9ffb04b0 P f4c5fa460e5a4a648bda699e263e0940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 } } peers { permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 } } peers { permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 } }
I20260810 07:27:52.160240 4117 consensus_queue.cc:260] T 927050ac821f4613b4ddb78d9ffb04b0 P 5ba31ad84ad84a7eb41bae121def8fc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 } } peers { permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 } } peers { permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 } }
I20260810 07:27:52.161020 4091 heartbeater.cc:503] Master 127.12.157.254:39709 was elected leader, sending a full tablet report...
I20260810 07:27:52.161345 4117 ts_tablet_manager.cc:1428] T 927050ac821f4613b4ddb78d9ffb04b0 P 5ba31ad84ad84a7eb41bae121def8fc6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 07:27:52.161459 4115 ts_tablet_manager.cc:1428] T 927050ac821f4613b4ddb78d9ffb04b0 P f4c5fa460e5a4a648bda699e263e0940: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260810 07:27:52.161792 3861 heartbeater.cc:503] Master 127.12.157.254:39709 was elected leader, sending a full tablet report...
I20260810 07:27:52.165668 3649 raft_consensus.cc:482] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:27:52.165836 3649 raft_consensus.cc:504] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 } } peers { permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 } } peers { permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 } }
I20260810 07:27:52.166852 3649 leader_election.cc:288] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ba31ad84ad84a7eb41bae121def8fc6 (127.12.157.195:42671), f4c5fa460e5a4a648bda699e263e0940 (127.12.157.194:36057)
I20260810 07:27:52.170966 3644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:52.167078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:36057 (local address 127.12.157.193:34571)
0810 07:27:52.167196 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:52.167212 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:52.167333 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:52.167908 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:52.167910 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:52.167927 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:52.168015 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:52.168020 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:52.168909 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:52.168913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:52.169756 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:52.169765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:52.170503 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:52.170506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:52.170612 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:52.170665 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 07:27:52.170740 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 07:27:52.170818 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":44}
I20260810 07:27:52.171097 4108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:52.167138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:40301 (local address 127.12.157.195:42671)
0810 07:27:52.167262 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:52.167267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:52.167526 (+ 259us) server_negotiation.cc:407] Connection header received
0810 07:27:52.167644 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:52.167647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:52.167684 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:52.167746 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:52.168094 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:52.168614 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:52.169912 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:52.170499 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:52.170614 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:52.170662 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:52.170890 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 07:27:52.170977 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 07:27:52.171331 4111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:52.167509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:34571 (local address 127.12.157.194:36057)
0810 07:27:52.167662 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:52.167666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:52.167681 (+ 15us) server_negotiation.cc:407] Connection header received
0810 07:27:52.167750 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:52.167753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:52.167801 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:52.167879 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:27:52.168156 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:52.168789 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:52.169882 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:52.170373 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:52.170508 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:52.170563 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:52.171139 (+ 576us) server_negotiation.cc:299] Negotiation successful
0810 07:27:52.171222 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 07:27:52.171429 4122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:52.167081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:42671 (local address 127.12.157.193:40301)
0810 07:27:52.167433 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:52.167444 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:52.167527 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:52.167778 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:52.167781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:52.167798 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:27:52.167974 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:52.167988 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:52.168741 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:52.168745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:52.169760 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:52.169770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:52.170655 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:52.170659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:52.170757 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:52.170776 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 07:27:52.171217 (+ 441us) client_negotiation.cc:240] Negotiation successful
0810 07:27:52.171288 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":86,"thread_start_us":113,"threads_started":1}
I20260810 07:27:52.171948 3789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "927050ac821f4613b4ddb78d9ffb04b0" candidate_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c5fa460e5a4a648bda699e263e0940" is_pre_election: true
I20260810 07:27:52.172015 4010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "927050ac821f4613b4ddb78d9ffb04b0" candidate_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" is_pre_election: true
I20260810 07:27:52.172204 3789 raft_consensus.cc:2454] T 927050ac821f4613b4ddb78d9ffb04b0 P f4c5fa460e5a4a648bda699e263e0940 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcf4b29ce0ac4dcc97cf34daed3fe769 in term 0.
I20260810 07:27:52.172346 4010 raft_consensus.cc:2454] T 927050ac821f4613b4ddb78d9ffb04b0 P 5ba31ad84ad84a7eb41bae121def8fc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcf4b29ce0ac4dcc97cf34daed3fe769 in term 0.
I20260810 07:27:52.172684 3448 leader_election.cc:302] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dcf4b29ce0ac4dcc97cf34daed3fe769, f4c5fa460e5a4a648bda699e263e0940; no voters:
I20260810 07:27:52.173024 3649 raft_consensus.cc:2781] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:27:52.173133 3649 raft_consensus.cc:482] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:27:52.173194 3649 raft_consensus.cc:3037] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:52.174295 3649 raft_consensus.cc:504] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 } } peers { permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 } } peers { permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 } }
I20260810 07:27:52.174680 3649 leader_election.cc:288] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [CANDIDATE]: Term 1 election: Requested vote from peers 5ba31ad84ad84a7eb41bae121def8fc6 (127.12.157.195:42671), f4c5fa460e5a4a648bda699e263e0940 (127.12.157.194:36057)
I20260810 07:27:52.175151 3789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "927050ac821f4613b4ddb78d9ffb04b0" candidate_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c5fa460e5a4a648bda699e263e0940"
I20260810 07:27:52.175143 4010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "927050ac821f4613b4ddb78d9ffb04b0" candidate_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ba31ad84ad84a7eb41bae121def8fc6"
I20260810 07:27:52.175283 3789 raft_consensus.cc:3037] T 927050ac821f4613b4ddb78d9ffb04b0 P f4c5fa460e5a4a648bda699e263e0940 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:52.175284 4010 raft_consensus.cc:3037] T 927050ac821f4613b4ddb78d9ffb04b0 P 5ba31ad84ad84a7eb41bae121def8fc6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:27:52.176637 4010 raft_consensus.cc:2454] T 927050ac821f4613b4ddb78d9ffb04b0 P 5ba31ad84ad84a7eb41bae121def8fc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcf4b29ce0ac4dcc97cf34daed3fe769 in term 1.
I20260810 07:27:52.176637 3789 raft_consensus.cc:2454] T 927050ac821f4613b4ddb78d9ffb04b0 P f4c5fa460e5a4a648bda699e263e0940 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcf4b29ce0ac4dcc97cf34daed3fe769 in term 1.
I20260810 07:27:52.177009 3448 leader_election.cc:302] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5ba31ad84ad84a7eb41bae121def8fc6, dcf4b29ce0ac4dcc97cf34daed3fe769, f4c5fa460e5a4a648bda699e263e0940; no voters:
I20260810 07:27:52.177193 3649 raft_consensus.cc:2781] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:27:52.177484 3649 raft_consensus.cc:686] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 1 LEADER]: Becoming Leader. State: Replica: dcf4b29ce0ac4dcc97cf34daed3fe769, State: Running, Role: LEADER
I20260810 07:27:52.177796 3649 consensus_queue.cc:237] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 } } peers { permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 } } peers { permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 } }
W20260810 07:27:52.180689 3638 tablet.cc:2369] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:27:52.180912 3318 catalog_manager.cc:5495] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcf4b29ce0ac4dcc97cf34daed3fe769 (127.12.157.193). New cstate: current_term: 1 leader_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 } health_report { overall_health: HEALTHY } } }
I20260810 07:27:52.203481 3652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:52.199892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37818 (local address 127.12.157.193:41497)
0810 07:27:52.200042 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:52.200046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:52.200059 (+ 13us) server_negotiation.cc:407] Connection header received
0810 07:27:52.200138 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:52.200140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:52.200194 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:52.200286 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:27:52.200656 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:52.201297 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:52.202093 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:52.202324 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:52.202402 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:52.202709 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:27:52.203150 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:27:52.203237 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:52.203318 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 07:27:52.203370 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 07:27:52.207612 3789 raft_consensus.cc:1261] T 927050ac821f4613b4ddb78d9ffb04b0 P f4c5fa460e5a4a648bda699e263e0940 [term 1 FOLLOWER]: Refusing update from remote peer dcf4b29ce0ac4dcc97cf34daed3fe769: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:27:52.207612 4010 raft_consensus.cc:1261] T 927050ac821f4613b4ddb78d9ffb04b0 P 5ba31ad84ad84a7eb41bae121def8fc6 [term 1 FOLLOWER]: Refusing update from remote peer dcf4b29ce0ac4dcc97cf34daed3fe769: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:27:52.208336 3649 consensus_queue.cc:1035] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:27:52.208562 4127 consensus_queue.cc:1035] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:27:52.215725 4131 mvcc.cc:205] Tried to move back new op lower bound from 7316876788555612160 to 7316876788442824704. Current Snapshot: MvccSnapshot[applied={T|T < 7316876788555612160}]
I20260810 07:27:52.219642 4132 mvcc.cc:205] Tried to move back new op lower bound from 7316876788555612160 to 7316876788442824704. Current Snapshot: MvccSnapshot[applied={T|T < 7316876788555612160}]
I20260810 07:27:52.224529 3318 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:47258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:27:52.224715 3318 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:47258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:27:52.227414 3318 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a: Sending DeleteTablet for 3 replicas of tablet 927050ac821f4613b4ddb78d9ffb04b0
I20260810 07:27:52.228209 3755 tablet_service.cc:1514] Processing DeleteTablet for tablet 927050ac821f4613b4ddb78d9ffb04b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:27:52 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:39682
I20260810 07:27:52.228273 3981 tablet_service.cc:1514] Processing DeleteTablet for tablet 927050ac821f4613b4ddb78d9ffb04b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:27:52 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:48064
I20260810 07:27:52.228524 3538 tablet_service.cc:1514] Processing DeleteTablet for tablet 927050ac821f4613b4ddb78d9ffb04b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:27:52 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:37808
I20260810 07:27:52.228699 3653 tablet_replica.cc:330] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769: stopping tablet replica
I20260810 07:27:52.228760 4147 tablet_replica.cc:330] T 927050ac821f4613b4ddb78d9ffb04b0 P 5ba31ad84ad84a7eb41bae121def8fc6: stopping tablet replica
I20260810 07:27:52.228806 3653 raft_consensus.cc:2229] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:27:52.229135 4147 raft_consensus.cc:2229] T 927050ac821f4613b4ddb78d9ffb04b0 P 5ba31ad84ad84a7eb41bae121def8fc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:27:52.229161 3653 raft_consensus.cc:2258] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:27:52.229467 4147 raft_consensus.cc:2258] T 927050ac821f4613b4ddb78d9ffb04b0 P 5ba31ad84ad84a7eb41bae121def8fc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:27:52.229940 3653 ts_tablet_manager.cc:1905] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:27:52.230243 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 3384
I20260810 07:27:52.240873 4147 ts_tablet_manager.cc:1905] T 927050ac821f4613b4ddb78d9ffb04b0 P 5ba31ad84ad84a7eb41bae121def8fc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:27:52.243963 4147 ts_tablet_manager.cc:1918] T 927050ac821f4613b4ddb78d9ffb04b0 P 5ba31ad84ad84a7eb41bae121def8fc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:27:52.244067 4147 log.cc:1196] T 927050ac821f4613b4ddb78d9ffb04b0 P 5ba31ad84ad84a7eb41bae121def8fc6: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/wal/wals/927050ac821f4613b4ddb78d9ffb04b0
I20260810 07:27:52.244405 4147 ts_tablet_manager.cc:1939] T 927050ac821f4613b4ddb78d9ffb04b0 P 5ba31ad84ad84a7eb41bae121def8fc6: Deleting consensus metadata
I20260810 07:27:52.244380 4146 tablet_replica.cc:330] T 927050ac821f4613b4ddb78d9ffb04b0 P f4c5fa460e5a4a648bda699e263e0940: stopping tablet replica
I20260810 07:27:52.244844 4146 raft_consensus.cc:2229] T 927050ac821f4613b4ddb78d9ffb04b0 P f4c5fa460e5a4a648bda699e263e0940 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:27:52.245257 4146 raft_consensus.cc:2258] T 927050ac821f4613b4ddb78d9ffb04b0 P f4c5fa460e5a4a648bda699e263e0940 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:27:52.245839 3295 catalog_manager.cc:4841] TS 5ba31ad84ad84a7eb41bae121def8fc6 (127.12.157.195:42671): tablet 927050ac821f4613b4ddb78d9ffb04b0 (table test-table [id=0d4a3861fc8645ed8ad9286715eed95e]) successfully deleted
I20260810 07:27:52.246994 4146 ts_tablet_manager.cc:1905] T 927050ac821f4613b4ddb78d9ffb04b0 P f4c5fa460e5a4a648bda699e263e0940: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:27:52.250547 4146 ts_tablet_manager.cc:1918] T 927050ac821f4613b4ddb78d9ffb04b0 P f4c5fa460e5a4a648bda699e263e0940: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:27:52.250658 4146 log.cc:1196] T 927050ac821f4613b4ddb78d9ffb04b0 P f4c5fa460e5a4a648bda699e263e0940: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/wal/wals/927050ac821f4613b4ddb78d9ffb04b0
W20260810 07:27:52.250949 3295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.157.193:41497 (error 108)
I20260810 07:27:52.251029 4146 ts_tablet_manager.cc:1939] T 927050ac821f4613b4ddb78d9ffb04b0 P f4c5fa460e5a4a648bda699e263e0940: Deleting consensus metadata
I20260810 07:27:52.251459 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 3646
I20260810 07:27:52.254559 4110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:52.254181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:41497 (local address 127.0.0.1:37824)
0810 07:27:52.254376 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:52.254447 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.193:41497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260810 07:27:52.254799 3295 catalog_manager.cc:4584] TS dcf4b29ce0ac4dcc97cf34daed3fe769 (127.12.157.193:41497): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 927050ac821f4613b4ddb78d9ffb04b0: Network error: Client connection negotiation failed: client connection to 127.12.157.193:41497: connect: Connection refused (error 111)
I20260810 07:27:52.263075 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 3866
I20260810 07:27:52.269913 4110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:52.263254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:36057 (local address 127.0.0.1:39696)
0810 07:27:52.269706 (+ 6452us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:52.269791 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.194:36057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6316}
I20260810 07:27:52.275687 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 3258
I20260810 07:27:52.290676 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:39709
--webserver_interface=127.12.157.254
--webserver_port=41799
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:46101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:39709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:27:52.408814 4150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:52.409127 4150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:52.409220 4150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:52.413350 4150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:27:52.413419 4150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:52.413439 4150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:52.413456 4150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:27:52.413472 4150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:27:52.418987 4150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:46101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:39709
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:39709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=41799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:27:52.420137 4150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:52.421228 4150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:52.430181 4161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:52.430590 4150 server_base.cc:1034] running on GCE node
W20260810 07:27:52.430188 4163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:52.430188 4165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:52.431360 4150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:52.432511 4150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:52.433701 4150 hybrid_clock.cc:648] HybridClock initialized: now 1786346872433682 us; error 33 us; skew 500 ppm
Aug 10 07:27:52 dist-test-slave-ffrd krb5kdc[2426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346872, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:52.436523 4150 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:27:52.437525 4150 webserver.cc:460] Webserver started at http://127.12.157.254:41799/ using document root <none> and password file <none>
I20260810 07:27:52.438123 4150 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:52.438220 4150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:52.442191 4150 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 07:27:52.444528 4172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:52.446142 4150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:27:52.446306 4150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "c72a6929623c409db8ae2e39c8aacf9a"
format_stamp: "Formatted at 2026-08-10 07:27:49 on dist-test-slave-ffrd"
server_key: "bf0dd917acaa08a98ffde4343e253ad4"
server_key_iv: "95f337f7d79b3b8a200c523842766a67"
server_key_version: "encryptionkey@0"
I20260810 07:27:52.446892 4150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:52.458887 4150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:52.462152 4150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:52.462369 4150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:52.470494 4150 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:39709
I20260810 07:27:52.470799 4253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:39709 every 8 connection(s)
I20260810 07:27:52.471733 4150 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:27:52.476178 4254 sys_catalog.cc:263] Verifying existing consensus state
I20260810 07:27:52.477844 4254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a: Bootstrap starting.
I20260810 07:27:52.478204 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 4150
I20260810 07:27:52.478684 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:41497
--local_ip_for_outbound_sockets=127.12.157.193
--tserver_master_addrs=127.12.157.254:39709
--webserver_port=46173
--webserver_interface=127.12.157.193
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:46101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:27:52.497406 4254 log.cc:824] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:52.508999 4254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 07:27:52.509418 4254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a: Bootstrap complete.
I20260810 07:27:52.513688 4254 raft_consensus.cc:348] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72a6929623c409db8ae2e39c8aacf9a" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39709 } }
I20260810 07:27:52.514146 4254 raft_consensus.cc:729] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c72a6929623c409db8ae2e39c8aacf9a, State: Initialized, Role: FOLLOWER
I20260810 07:27:52.514649 4254 consensus_queue.cc:260] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72a6929623c409db8ae2e39c8aacf9a" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39709 } }
I20260810 07:27:52.514793 4254 raft_consensus.cc:388] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:27:52.514868 4254 raft_consensus.cc:482] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:27:52.515000 4254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [term 2 FOLLOWER]: Advancing to term 3
I20260810 07:27:52.516953 4254 raft_consensus.cc:504] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72a6929623c409db8ae2e39c8aacf9a" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39709 } }
I20260810 07:27:52.517338 4254 leader_election.cc:302] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c72a6929623c409db8ae2e39c8aacf9a; no voters:
I20260810 07:27:52.517693 4254 leader_election.cc:288] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 07:27:52.517792 4259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [term 3 FOLLOWER]: Leader election won for term 3
I20260810 07:27:52.517980 4259 raft_consensus.cc:686] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [term 3 LEADER]: Becoming Leader. State: Replica: c72a6929623c409db8ae2e39c8aacf9a, State: Running, Role: LEADER
I20260810 07:27:52.518288 4259 consensus_queue.cc:237] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72a6929623c409db8ae2e39c8aacf9a" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39709 } }
I20260810 07:27:52.519003 4254 sys_catalog.cc:564] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:27:52.519712 4260 sys_catalog.cc:455] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c72a6929623c409db8ae2e39c8aacf9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72a6929623c409db8ae2e39c8aacf9a" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39709 } } }
I20260810 07:27:52.520141 4260 sys_catalog.cc:458] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:52.520041 4262 sys_catalog.cc:455] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c72a6929623c409db8ae2e39c8aacf9a. Latest consensus state: current_term: 3 leader_uuid: "c72a6929623c409db8ae2e39c8aacf9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c72a6929623c409db8ae2e39c8aacf9a" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 39709 } } }
I20260810 07:27:52.520530 4262 sys_catalog.cc:458] T 00000000000000000000000000000000 P c72a6929623c409db8ae2e39c8aacf9a [sys.catalog]: This master's current role is: LEADER
I20260810 07:27:52.520843 4276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:27:52.526333 4276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:27:52.527040 4276 catalog_manager.cc:1259] Loaded cluster ID: d721cb79fc0641e0b89ce9aae9674901
I20260810 07:27:52.527122 4276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 07:27:52.527959 4276 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fdf033259ad google::LogMessage::Fail() at ??:0
@ 0x7fdf0332888a google::LogMessage::SendToLog() at ??:0
@ 0x7fdf033254de google::LogMessage::Flush() at ??:0
@ 0x7fdf03326a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdf0dc64fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fdf0dc65d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fdf0dc62579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fdf0dca6640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fdf055773d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdf05577c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fdf055793f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fdf05568512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdf06bac6db start_thread at ??:0
@ 0x7fdf01e2361f clone at ??:0
W20260810 07:27:52.613139 4256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:52.613371 4256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:52.613415 4256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:52.617815 4256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:52.617902 4256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:52.618021 4256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:27:52.623492 4256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:46101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:41497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=46173
--tserver_master_addrs=127.12.157.254:39709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:27:52.624610 4256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:52.625743 4256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:52.635840 4325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:52.635852 4324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:52.636119 4327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:52.636201 4256 server_base.cc:1034] running on GCE node
I20260810 07:27:52.636813 4256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:52.637653 4256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:52.638868 4256 hybrid_clock.cc:648] HybridClock initialized: now 1786346872638824 us; error 51 us; skew 500 ppm
Aug 10 07:27:52 dist-test-slave-ffrd krb5kdc[2426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346872, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:52.642499 4256 init.cc:377] Logged in from keytab as kudu/127.12.157.193@KRBTEST.COM (short username kudu)
I20260810 07:27:52.643659 4256 webserver.cc:460] Webserver started at http://127.12.157.193:46173/ using document root <none> and password file <none>
I20260810 07:27:52.644394 4256 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:52.644520 4256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:52.649768 4256 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 07:27:52.652637 4336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:52.654346 4256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:27:52.654506 4256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769"
format_stamp: "Formatted at 2026-08-10 07:27:50 on dist-test-slave-ffrd"
server_key: "b6b72b66603fd0e3ccedf473e1313806"
server_key_iv: "58712e49785086b3d04a96752a52aca4"
server_key_version: "encryptionkey@0"
I20260810 07:27:52.655231 4256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:52.682168 4256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:52.686818 4256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:52.687063 4256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:52.687911 4256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:52.689369 4346 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 07:27:52.694181 4256 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 07:27:52.694269 4256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260810 07:27:52.694339 4256 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 07:27:52.696673 4256 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 07:27:52.696738 4256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 07:27:52.697124 4346 tablet_bootstrap.cc:492] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769: Bootstrap starting.
I20260810 07:27:52.704366 4346 log.cc:824] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769: Log is configured to *not* fsync() on all Append() calls
I20260810 07:27:52.708585 4256 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:41497
I20260810 07:27:52.708619 4529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:41497 every 8 connection(s)
I20260810 07:27:52.709847 4256 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:27:52.713676 4346 tablet_bootstrap.cc:492] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 07:27:52.714141 4346 tablet_bootstrap.cc:492] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769: Bootstrap complete.
I20260810 07:27:52.714493 4346 ts_tablet_manager.cc:1397] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.003s
I20260810 07:27:52.716815 4346 raft_consensus.cc:348] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 } } peers { permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 } } peers { permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 } }
I20260810 07:27:52.717412 4346 raft_consensus.cc:729] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcf4b29ce0ac4dcc97cf34daed3fe769, State: Initialized, Role: FOLLOWER
I20260810 07:27:52.717476 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 4256
I20260810 07:27:52.717932 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:36057
--local_ip_for_outbound_sockets=127.12.157.194
--tserver_master_addrs=127.12.157.254:39709
--webserver_port=35389
--webserver_interface=127.12.157.194
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:46101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:27:52.717941 4346 consensus_queue.cc:260] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 } } peers { permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 } } peers { permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 } }
I20260810 07:27:52.719040 4346 ts_tablet_manager.cc:1428] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260810 07:27:52.777987 4535 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:27:52.712188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.193:58397)
0810 07:27:52.712795 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:52.712852 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:52.713974 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:52.777016 (+ 63042us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":165,"threads_started":1}
W20260810 07:27:52.778575 4530 heartbeater.cc:644] Failed to heartbeat to 127.12.157.254:39709 (0 consecutive failures): Network error: Failed to ping master at 127.12.157.254:39709: Client connection negotiation failed: client connection to 127.12.157.254:39709: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 07:27:52.779403 4535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:52.779116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.193:52453)
0810 07:27:52.779259 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:52.779338 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 07:27:52.780282 4535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:52.780067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.193:32877)
0810 07:27:52.780187 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:52.780231 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 07:27:52.780575 4530 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 07:27:52.838018 4537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:52.838297 4537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:52.838372 4537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:52.842300 4537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:52.842386 4537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:52.842482 4537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:27:52.847700 4537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:46101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:36057
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=35389
--tserver_master_addrs=127.12.157.254:39709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:27:52.848716 4537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:52.849740 4537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:52.859197 4553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:52.859179 4550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:52.859201 4549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:52.859494 4537 server_base.cc:1034] running on GCE node
I20260810 07:27:52.859853 4537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:52.860452 4537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:52.861645 4537 hybrid_clock.cc:648] HybridClock initialized: now 1786346872861621 us; error 42 us; skew 500 ppm
Aug 10 07:27:52 dist-test-slave-ffrd krb5kdc[2426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346872, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:52.864435 4537 init.cc:377] Logged in from keytab as kudu/127.12.157.194@KRBTEST.COM (short username kudu)
I20260810 07:27:52.865370 4537 webserver.cc:460] Webserver started at http://127.12.157.194:35389/ using document root <none> and password file <none>
I20260810 07:27:52.865918 4537 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:52.866017 4537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:52.869767 4537 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:27:52.871994 4565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:52.873258 4537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 07:27:52.873391 4537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "f4c5fa460e5a4a648bda699e263e0940"
format_stamp: "Formatted at 2026-08-10 07:27:50 on dist-test-slave-ffrd"
server_key: "c0397f75dfbe44945d29906535c8fb88"
server_key_iv: "31ca7d568640b81076c0f66897d7ffbc"
server_key_version: "encryptionkey@0"
I20260810 07:27:52.873932 4537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:52.889264 4537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:52.892554 4537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:52.892781 4537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:52.893461 4537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:52.894330 4537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:52.894404 4537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:52.894479 4537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:52.894523 4537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:52.904093 4537 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:36057
I20260810 07:27:52.904125 4762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:36057 every 8 connection(s)
I20260810 07:27:52.905143 4537 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:27:52.905865 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 4537
I20260810 07:27:52.906332 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:42671
--local_ip_for_outbound_sockets=127.12.157.195
--tserver_master_addrs=127.12.157.254:39709
--webserver_port=46713
--webserver_interface=127.12.157.195
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:46101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:27:52.909602 4771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:52.907510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.194:43553)
0810 07:27:52.908016 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:52.908150 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":145,"threads_started":1}
W20260810 07:27:52.910166 4764 heartbeater.cc:644] Failed to heartbeat to 127.12.157.254:39709 (0 consecutive failures): Network error: Failed to ping master at 127.12.157.254:39709: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
I20260810 07:27:52.910944 4771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:52.910717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.194:35321)
0810 07:27:52.910839 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:52.910888 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 07:27:52.911687 4771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:52.911457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.194:48529)
0810 07:27:52.911582 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:52.911632 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260810 07:27:52.911887 4764 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 07:27:53.024030 4773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:27:53.024331 4773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:27:53.024411 4773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:27:53.028538 4773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:27:53.028626 4773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:27:53.028715 4773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:27:53.034366 4773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:46101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:42671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=46713
--tserver_master_addrs=127.12.157.254:39709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:27:53.035436 4773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:27:53.036437 4773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:27:53.046325 4786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:27:53.046345 4783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:53.046418 4773 server_base.cc:1034] running on GCE node
W20260810 07:27:53.046643 4789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:27:53.047031 4773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:27:53.047734 4773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:27:53.048934 4773 hybrid_clock.cc:648] HybridClock initialized: now 1786346873048898 us; error 48 us; skew 500 ppm
Aug 10 07:27:53 dist-test-slave-ffrd krb5kdc[2426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346873, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:27:53.051797 4773 init.cc:377] Logged in from keytab as kudu/127.12.157.195@KRBTEST.COM (short username kudu)
I20260810 07:27:53.052789 4773 webserver.cc:460] Webserver started at http://127.12.157.195:46713/ using document root <none> and password file <none>
I20260810 07:27:53.053436 4773 fs_manager.cc:359] Metadata directory not provided
I20260810 07:27:53.053543 4773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:27:53.057793 4773 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 07:27:53.059903 4803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:53.061331 4773 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 07:27:53.061470 4773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "5ba31ad84ad84a7eb41bae121def8fc6"
format_stamp: "Formatted at 2026-08-10 07:27:50 on dist-test-slave-ffrd"
server_key: "8d14b162bd5a5b6a15b81be775336175"
server_key_iv: "72e1b38a26936d3b2ec5a90c620d2883"
server_key_version: "encryptionkey@0"
I20260810 07:27:53.062031 4773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:27:53.079039 4773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:27:53.082355 4773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:27:53.082561 4773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:27:53.083166 4773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:27:53.083992 4773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:27:53.084039 4773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:53.084110 4773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:27:53.084151 4773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:27:53.093068 4773 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:42671
I20260810 07:27:53.093165 4998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:42671 every 8 connection(s)
I20260810 07:27:53.094113 4773 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:27:53.094828 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 4773
I20260810 07:27:53.098280 5008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:53.096507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.195:47501)
0810 07:27:53.097013 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:53.097141 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":107,"threads_started":1}
W20260810 07:27:53.098738 4999 heartbeater.cc:644] Failed to heartbeat to 127.12.157.254:39709 (0 consecutive failures): Network error: Failed to ping master at 127.12.157.254:39709: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
I20260810 07:27:53.099319 5008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:53.099118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.195:51719)
0810 07:27:53.099219 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:53.099266 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 07:27:53.099960 5008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:53.099781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.195:41947)
0810 07:27:53.099870 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:53.099910 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 07:27:53.100186 4999 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 07:27:53.781723 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:53.781242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.193:55939)
0810 07:27:53.781560 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:53.781620 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":80,"threads_started":1}
I20260810 07:27:53.912930 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:53.912462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.194:54807)
0810 07:27:53.912772 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:53.912826 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":85,"threads_started":1}
I20260810 07:27:54.027446 5011 raft_consensus.cc:482] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:27:54.027688 5011 raft_consensus.cc:504] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 } } peers { permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 } } peers { permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 } }
I20260810 07:27:54.028860 5011 leader_election.cc:288] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ba31ad84ad84a7eb41bae121def8fc6 (127.12.157.195:42671), f4c5fa460e5a4a648bda699e263e0940 (127.12.157.194:36057)
Aug 10 07:27:54 dist-test-slave-ffrd krb5kdc[2426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346872, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for kudu/127.12.157.195@KRBTEST.COM
Aug 10 07:27:54 dist-test-slave-ffrd krb5kdc[2426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346872, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for kudu/127.12.157.194@KRBTEST.COM
I20260810 07:27:54.047786 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:54.029179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:36057 (local address 127.12.157.193:57271)
0810 07:27:54.029324 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:54.029343 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:54.029485 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:54.031308 (+ 1823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:54.031314 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:54.031821 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:54.032343 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:54.032357 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:54.033753 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:54.033756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:54.034861 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:54.034871 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:54.035596 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:54.035600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:54.035891 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:54.040678 (+ 4787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:54.040696 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:54.043079 (+ 2383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:54.045382 (+ 2303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:54.045387 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:54.045389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:54.045610 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:54.045976 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:54.045980 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:54.045982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:54.046073 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:54.046734 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:54.046741 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:54.047001 (+ 260us) client_negotiation.cc:769] Sending connection context
0810 07:27:54.047197 (+ 196us) client_negotiation.cc:240] Negotiation successful
0810 07:27:54.047562 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":248,"spinlock_wait_cycles":45440}
I20260810 07:27:54.047787 5014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:54.029363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:57271 (local address 127.12.157.194:36057)
0810 07:27:54.029803 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:54.029840 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:54.029874 (+ 34us) server_negotiation.cc:407] Connection header received
0810 07:27:54.030587 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:54.030593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:54.030952 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:54.031914 (+ 962us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:54.032525 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:54.033587 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:54.034989 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:54.035464 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:54.035647 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:54.043311 (+ 7664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:54.043317 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:54.043329 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:54.043359 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:54.045256 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:54.045741 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:54.045746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:54.045751 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:54.045869 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:54.046169 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:54.046174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:54.046176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:54.046608 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:54.046760 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:54.047313 (+ 553us) server_negotiation.cc:299] Negotiation successful
0810 07:27:54.047585 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":114,"threads_started":1}
I20260810 07:27:54.047835 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:54.029371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:34837 (local address 127.12.157.195:42671)
0810 07:27:54.029779 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0810 07:27:54.029797 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 07:27:54.029848 (+ 51us) server_negotiation.cc:407] Connection header received
0810 07:27:54.030545 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:27:54.030551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:27:54.030959 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:27:54.031256 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:27:54.032538 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:54.033615 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:54.035011 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:27:54.035474 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:27:54.035649 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:54.043034 (+ 7385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:27:54.043040 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:27:54.043053 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:27:54.043086 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:27:54.045062 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:54.045684 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:54.045689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:54.045693 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:54.045810 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:27:54.046168 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:27:54.046172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:27:54.046174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:27:54.046609 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:27:54.046760 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 07:27:54.047319 (+ 559us) server_negotiation.cc:299] Negotiation successful
0810 07:27:54.047622 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":91,"threads_started":1}
I20260810 07:27:54.047827 5012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:54.029179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:42671 (local address 127.12.157.193:34837)
0810 07:27:54.029716 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:54.029730 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:27:54.029830 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:27:54.031305 (+ 1475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:27:54.031311 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:27:54.031897 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:27:54.032342 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:54.032357 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:54.033754 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:54.033758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:54.034867 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:27:54.034877 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:27:54.035605 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:27:54.035608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:27:54.035889 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:27:54.040666 (+ 4777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:27:54.040705 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:27:54.042732 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:27:54.045191 (+ 2459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:54.045198 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:54.045212 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:54.045544 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:54.045914 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:27:54.045918 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:27:54.045920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:27:54.046066 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:27:54.046734 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:27:54.046741 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:27:54.046998 (+ 257us) client_negotiation.cc:769] Sending connection context
0810 07:27:54.047198 (+ 200us) client_negotiation.cc:240] Negotiation successful
0810 07:27:54.047650 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":497,"spinlock_wait_cycles":20352,"thread_start_us":80,"threads_started":1}
W20260810 07:27:54.050581 4340 leader_election.cc:341] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ba31ad84ad84a7eb41bae121def8fc6 (127.12.157.195:42671): Not found: Tablet not found: 927050ac821f4613b4ddb78d9ffb04b0
W20260810 07:27:54.050771 4338 leader_election.cc:341] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f4c5fa460e5a4a648bda699e263e0940 (127.12.157.194:36057): Not found: Tablet not found: 927050ac821f4613b4ddb78d9ffb04b0
I20260810 07:27:54.050901 4338 leader_election.cc:302] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dcf4b29ce0ac4dcc97cf34daed3fe769; no voters: 5ba31ad84ad84a7eb41bae121def8fc6, f4c5fa460e5a4a648bda699e263e0940
I20260810 07:27:54.051201 5011 raft_consensus.cc:2726] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 07:27:54.101264 5016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:54.100760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.195:51013)
0810 07:27:54.101094 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:54.101161 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":91,"threads_started":1}
I20260810 07:27:54.783057 5019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:54.782605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.193:60471)
0810 07:27:54.782882 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:54.782955 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":83,"threads_started":1}
I20260810 07:27:54.914256 5020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:54.913842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.194:53261)
0810 07:27:54.914099 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:54.914160 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":80,"threads_started":1}
I20260810 07:27:55.102564 5022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:55.102074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.195:42503)
0810 07:27:55.102374 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:55.102454 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":77,"threads_started":1}
I20260810 07:27:55.784271 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:55.783872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.193:53035)
0810 07:27:55.784121 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:55.784174 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":78,"threads_started":1}
I20260810 07:27:55.875924 5026 raft_consensus.cc:482] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:27:55.876065 5026 raft_consensus.cc:504] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 } } peers { permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 } } peers { permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 } }
I20260810 07:27:55.876494 5026 leader_election.cc:288] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ba31ad84ad84a7eb41bae121def8fc6 (127.12.157.195:42671), f4c5fa460e5a4a648bda699e263e0940 (127.12.157.194:36057)
W20260810 07:27:55.877446 4338 leader_election.cc:341] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f4c5fa460e5a4a648bda699e263e0940 (127.12.157.194:36057): Not found: Tablet not found: 927050ac821f4613b4ddb78d9ffb04b0
W20260810 07:27:55.877543 4340 leader_election.cc:341] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ba31ad84ad84a7eb41bae121def8fc6 (127.12.157.195:42671): Not found: Tablet not found: 927050ac821f4613b4ddb78d9ffb04b0
I20260810 07:27:55.877610 4340 leader_election.cc:302] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dcf4b29ce0ac4dcc97cf34daed3fe769; no voters: 5ba31ad84ad84a7eb41bae121def8fc6, f4c5fa460e5a4a648bda699e263e0940
I20260810 07:27:55.877835 5026 raft_consensus.cc:2726] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 07:27:55.915439 5029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:55.915007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.194:47613)
0810 07:27:55.915270 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:55.915338 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":76,"threads_started":1}
I20260810 07:27:56.103762 5030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:56.103322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.195:45353)
0810 07:27:56.103609 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:56.103669 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":76,"threads_started":1}
I20260810 07:27:56.785744 5032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:56.785253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.193:43307)
0810 07:27:56.785573 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:56.785639 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":102,"threads_started":1}
I20260810 07:27:56.916755 5034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:56.916339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.194:50897)
0810 07:27:56.916597 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:56.916657 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":78,"threads_started":1}
I20260810 07:27:57.105181 5035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:57.104647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.195:57959)
0810 07:27:57.104990 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:57.105069 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":108,"threads_started":1}
I20260810 07:27:57.787432 5036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:57.786806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.193:55833)
0810 07:27:57.787226 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:57.787316 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":83,"threads_started":1}
I20260810 07:27:57.918322 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:57.917751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.194:37307)
0810 07:27:57.918142 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:57.918209 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":161,"threads_started":1}
I20260810 07:27:58.106462 5039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:58.106017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.195:40923)
0810 07:27:58.106303 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:58.106367 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":100,"threads_started":1}
I20260810 07:27:58.684386 5043 raft_consensus.cc:482] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:27:58.684566 5043 raft_consensus.cc:504] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 } } peers { permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 } } peers { permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 } }
I20260810 07:27:58.685082 5043 leader_election.cc:288] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ba31ad84ad84a7eb41bae121def8fc6 (127.12.157.195:42671), f4c5fa460e5a4a648bda699e263e0940 (127.12.157.194:36057)
W20260810 07:27:58.685986 4338 leader_election.cc:341] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f4c5fa460e5a4a648bda699e263e0940 (127.12.157.194:36057): Not found: Tablet not found: 927050ac821f4613b4ddb78d9ffb04b0
W20260810 07:27:58.686094 4340 leader_election.cc:341] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ba31ad84ad84a7eb41bae121def8fc6 (127.12.157.195:42671): Not found: Tablet not found: 927050ac821f4613b4ddb78d9ffb04b0
I20260810 07:27:58.686165 4340 leader_election.cc:302] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dcf4b29ce0ac4dcc97cf34daed3fe769; no voters: 5ba31ad84ad84a7eb41bae121def8fc6, f4c5fa460e5a4a648bda699e263e0940
I20260810 07:27:58.686307 5043 raft_consensus.cc:2726] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 07:27:58.788619 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:58.788134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.193:34527)
0810 07:27:58.788458 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:58.788517 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":106,"threads_started":1}
I20260810 07:27:58.919565 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:58.919107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.194:36139)
0810 07:27:58.919411 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:58.919468 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":97,"threads_started":1}
I20260810 07:27:59.107733 5050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:59.107290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.195:44981)
0810 07:27:59.107577 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:59.107638 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":97,"threads_started":1}
I20260810 07:27:59.790124 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:59.789522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.193:58863)
0810 07:27:59.789923 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:59.790020 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":102,"threads_started":1}
I20260810 07:27:59.921054 5057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:27:59.920432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.194:44929)
0810 07:27:59.920789 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0810 07:27:59.920865 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":86,"threads_started":1}
I20260810 07:28:00.109295 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:00.108583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.195:46731)
0810 07:28:00.109120 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:00.109189 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":99,"threads_started":1}
I20260810 07:28:00.267256 5059 raft_consensus.cc:482] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:28:00.267536 5059 raft_consensus.cc:504] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 } } peers { permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 } } peers { permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 } }
I20260810 07:28:00.268036 5059 leader_election.cc:288] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ba31ad84ad84a7eb41bae121def8fc6 (127.12.157.195:42671), f4c5fa460e5a4a648bda699e263e0940 (127.12.157.194:36057)
W20260810 07:28:00.268882 4340 leader_election.cc:341] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ba31ad84ad84a7eb41bae121def8fc6 (127.12.157.195:42671): Not found: Tablet not found: 927050ac821f4613b4ddb78d9ffb04b0
W20260810 07:28:00.269035 4338 leader_election.cc:341] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f4c5fa460e5a4a648bda699e263e0940 (127.12.157.194:36057): Not found: Tablet not found: 927050ac821f4613b4ddb78d9ffb04b0
I20260810 07:28:00.269132 4338 leader_election.cc:302] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dcf4b29ce0ac4dcc97cf34daed3fe769; no voters: 5ba31ad84ad84a7eb41bae121def8fc6, f4c5fa460e5a4a648bda699e263e0940
I20260810 07:28:00.269309 5059 raft_consensus.cc:2726] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 07:28:00.791378 5061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:00.790950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.193:45931)
0810 07:28:00.791212 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:00.791274 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":80,"threads_started":1}
I20260810 07:28:00.922354 5062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:00.921896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.194:59925)
0810 07:28:00.922181 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:00.922252 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":78,"threads_started":1}
I20260810 07:28:01.110724 5063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:01.110189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.195:39281)
0810 07:28:01.110548 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:01.110612 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":88,"threads_started":1}
I20260810 07:28:01.792718 5064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:01.792207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.193:56069)
0810 07:28:01.792546 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:01.792615 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":85,"threads_started":1}
I20260810 07:28:01.923725 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:01.923254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.194:56525)
0810 07:28:01.923558 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:01.923622 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":116,"threads_started":1}
I20260810 07:28:02.111958 5067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:02.111528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.195:50367)
0810 07:28:02.111791 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:02.111849 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":78,"threads_started":1}
I20260810 07:28:02.794039 5068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:02.793622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.193:58927)
0810 07:28:02.793867 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:02.793923 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":89,"threads_started":1}
I20260810 07:28:02.925065 5069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:02.924566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.194:55227)
0810 07:28:02.924839 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:02.924897 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":79,"threads_started":1}
I20260810 07:28:03.113238 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:03.112744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.195:34957)
0810 07:28:03.113074 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:03.113140 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":93,"threads_started":1}
I20260810 07:28:03.795321 5074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:03.794874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.193:36559)
0810 07:28:03.795152 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:03.795217 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":77,"threads_started":1}
I20260810 07:28:03.926402 5075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:03.925928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.194:57643)
0810 07:28:03.926232 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:03.926295 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":82,"threads_started":1}
I20260810 07:28:04.114619 5076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:04.114127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.195:37561)
0810 07:28:04.114439 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:04.114513 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":85,"threads_started":1}
I20260810 07:28:04.796802 5077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:04.796295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.193:42663)
0810 07:28:04.796617 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:04.796699 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":89,"threads_started":1}
I20260810 07:28:04.852860 5080 raft_consensus.cc:482] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:28:04.853052 5080 raft_consensus.cc:504] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ba31ad84ad84a7eb41bae121def8fc6" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 42671 } } peers { permanent_uuid: "f4c5fa460e5a4a648bda699e263e0940" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 36057 } } peers { permanent_uuid: "dcf4b29ce0ac4dcc97cf34daed3fe769" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 41497 } }
I20260810 07:28:04.853518 5080 leader_election.cc:288] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ba31ad84ad84a7eb41bae121def8fc6 (127.12.157.195:42671), f4c5fa460e5a4a648bda699e263e0940 (127.12.157.194:36057)
W20260810 07:28:04.854401 4340 leader_election.cc:341] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ba31ad84ad84a7eb41bae121def8fc6 (127.12.157.195:42671): Not found: Tablet not found: 927050ac821f4613b4ddb78d9ffb04b0
W20260810 07:28:04.854555 4338 leader_election.cc:341] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f4c5fa460e5a4a648bda699e263e0940 (127.12.157.194:36057): Not found: Tablet not found: 927050ac821f4613b4ddb78d9ffb04b0
I20260810 07:28:04.854635 4338 leader_election.cc:302] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dcf4b29ce0ac4dcc97cf34daed3fe769; no voters: 5ba31ad84ad84a7eb41bae121def8fc6, f4c5fa460e5a4a648bda699e263e0940
I20260810 07:28:04.854765 5080 raft_consensus.cc:2726] T 927050ac821f4613b4ddb78d9ffb04b0 P dcf4b29ce0ac4dcc97cf34daed3fe769 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 07:28:04.927829 5081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:04.927307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.194:47741)
0810 07:28:04.927647 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:04.927717 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":98,"threads_started":1}
I20260810 07:28:05.116009 5083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:05.115562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.195:50097)
0810 07:28:05.115828 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:05.115889 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":80,"threads_started":1}
I20260810 07:28:05.798128 5084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:05.797694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.193:38811)
0810 07:28:05.797969 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:05.798027 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":79,"threads_started":1}
I20260810 07:28:05.929008 5085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:05.928564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.194:57913)
0810 07:28:05.928820 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:05.928880 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":75,"threads_started":1}
I20260810 07:28:06.117439 5086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:06.116916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.195:56115)
0810 07:28:06.117263 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:06.117323 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":77,"threads_started":1}
I20260810 07:28:06.799528 5087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:06.798990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.193:43331)
0810 07:28:06.799287 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:06.799397 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":116,"threads_started":1}
I20260810 07:28:06.930248 5088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:06.929777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.194:39735)
0810 07:28:06.930082 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:06.930145 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":82,"threads_started":1}
I20260810 07:28:07.118974 5089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:07.118323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.195:45233)
0810 07:28:07.118779 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:07.118852 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":126,"threads_started":1}
I20260810 07:28:07.801354 5090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:07.800792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.193:60537)
0810 07:28:07.801113 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:07.801207 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":122,"threads_started":1}
I20260810 07:28:07.931643 5091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:07.931200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.194:43213)
0810 07:28:07.931481 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:07.931543 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":78,"threads_started":1}
I20260810 07:28:08.098177 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 4256
I20260810 07:28:08.111847 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 4537
I20260810 07:28:08.120637 5095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:08.120061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:39709 (local address 127.12.157.195:58421)
0810 07:28:08.120396 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:08.120483 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:39709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":73,"threads_started":1}
I20260810 07:28:08.124727 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 4773
2026-08-10T07:28:08Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19751 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:28:08 dist-test-slave-ffrd krb5kdc[5100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:28:08 dist-test-slave-ffrd krb5kdc[5100](info): set up 2 sockets
Aug 10 07:28:08 dist-test-slave-ffrd krb5kdc[5100](info): commencing operation
krb5kdc: starting...
W20260810 07:28:09.410085 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.252s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:28:09 dist-test-slave-ffrd krb5kdc[5100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:28:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:28:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:28:09.553996 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:43331
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:34025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:43331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:09.673883 5125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:09.674207 5125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:09.674291 5125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:09.678313 5125 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 07:28:09.678387 5125 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 07:28:09.678519 5125 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 07:28:09.678619 5125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:28:09.678669 5125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:09.678710 5125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:09.678745 5125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:28:09.678779 5125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:28:09.684672 5125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:34025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:43331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:43331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:28:09.685776 5125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:09.686806 5125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:09.695878 5140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:09.695973 5134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:09.695945 5136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:09.696369 5125 server_base.cc:1034] running on GCE node
I20260810 07:28:09.697021 5125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:09.698163 5125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:09.699364 5125 hybrid_clock.cc:648] HybridClock initialized: now 1786346889699329 us; error 42 us; skew 500 ppm
Aug 10 07:28:09 dist-test-slave-ffrd krb5kdc[5100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346889, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:09.706902 5125 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:28:09.707890 5125 webserver.cc:460] Webserver started at http://127.12.157.254:42839/ using document root <none> and password file <none>
I20260810 07:28:09.708493 5125 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:09.708590 5125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:09.708835 5125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:09.711122 5125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "3bb300bc5b3c41a1811369fec9238ce7"
format_stamp: "Formatted at 2026-08-10 07:28:09 on dist-test-slave-ffrd"
server_key: "cd65317d4fb8261f01ed216c32c2f3f5"
server_key_iv: "1a3a3db574c0d3c3e7b35f1370f467a4"
server_key_version: "encryptionkey@0"
I20260810 07:28:09.711758 5125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "3bb300bc5b3c41a1811369fec9238ce7"
format_stamp: "Formatted at 2026-08-10 07:28:09 on dist-test-slave-ffrd"
server_key: "cd65317d4fb8261f01ed216c32c2f3f5"
server_key_iv: "1a3a3db574c0d3c3e7b35f1370f467a4"
server_key_version: "encryptionkey@0"
I20260810 07:28:09.715572 5125 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 07:28:09.718034 5154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:09.719291 5125 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:28:09.719471 5125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "3bb300bc5b3c41a1811369fec9238ce7"
format_stamp: "Formatted at 2026-08-10 07:28:09 on dist-test-slave-ffrd"
server_key: "cd65317d4fb8261f01ed216c32c2f3f5"
server_key_iv: "1a3a3db574c0d3c3e7b35f1370f467a4"
server_key_version: "encryptionkey@0"
I20260810 07:28:09.719591 5125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:09.735320 5125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:09.738615 5125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:09.738837 5125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:09.746934 5125 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:43331
I20260810 07:28:09.747238 5244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:43331 every 8 connection(s)
I20260810 07:28:09.747995 5125 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:28:09.750952 5245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:09.752226 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 5125
I20260810 07:28:09.752380 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:28:09.752677 12919 external_mini_cluster.cc:1428] Setting key e74f1b5765920c352bc70b4618e8d9df
I20260810 07:28:09.757480 5245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7: Bootstrap starting.
I20260810 07:28:09.760330 5245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:09.761196 5245 log.cc:824] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7: Log is configured to *not* fsync() on all Append() calls
Aug 10 07:28:09 dist-test-slave-ffrd krb5kdc[5100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:28:09.763352 5245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7: No bootstrap required, opened a new log
I20260810 07:28:09.767280 5245 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb300bc5b3c41a1811369fec9238ce7" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 43331 } }
I20260810 07:28:09.767562 5245 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:09.767645 5245 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bb300bc5b3c41a1811369fec9238ce7, State: Initialized, Role: FOLLOWER
I20260810 07:28:09.767982 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:09.754174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46958 (local address 127.12.157.254:43331)
0810 07:28:09.754594 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:09.754619 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:09.754655 (+ 36us) server_negotiation.cc:407] Connection header received
0810 07:28:09.755301 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:09.755318 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:09.755626 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:09.755901 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:09.756874 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:09.758018 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:09.759216 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:09.759759 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:09.759901 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:09.762766 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:09.762771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:09.762786 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:09.762813 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:09.764588 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:09.765100 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:09.765106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:09.765110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:09.765227 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:09.765706 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:09.765711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:09.765713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:09.766169 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:09.766423 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:09.766729 (+ 306us) server_negotiation.cc:299] Negotiation successful
0810 07:28:09.767048 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":148,"threads_started":1}
I20260810 07:28:09.768143 5245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb300bc5b3c41a1811369fec9238ce7" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 43331 } }
I20260810 07:28:09.768316 5245 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:28:09.768412 5245 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:28:09.768687 5245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:09.770020 5245 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb300bc5b3c41a1811369fec9238ce7" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 43331 } }
I20260810 07:28:09.770395 5245 leader_election.cc:302] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bb300bc5b3c41a1811369fec9238ce7; no voters:
I20260810 07:28:09.770776 5245 leader_election.cc:288] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:28:09.770901 5253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:28:09.771162 5253 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [term 1 LEADER]: Becoming Leader. State: Replica: 3bb300bc5b3c41a1811369fec9238ce7, State: Running, Role: LEADER
I20260810 07:28:09.771502 5253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb300bc5b3c41a1811369fec9238ce7" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 43331 } }
I20260810 07:28:09.772100 5245 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:28:09.773325 5255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bb300bc5b3c41a1811369fec9238ce7. Latest consensus state: current_term: 1 leader_uuid: "3bb300bc5b3c41a1811369fec9238ce7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb300bc5b3c41a1811369fec9238ce7" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 43331 } } }
I20260810 07:28:09.773473 5255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:09.773620 5254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bb300bc5b3c41a1811369fec9238ce7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb300bc5b3c41a1811369fec9238ce7" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 43331 } } }
I20260810 07:28:09.773779 5254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:09.774116 5267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:28:09.777452 5267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:28:09.783064 5267 catalog_manager.cc:1347] Generated new cluster ID: 66c0b25916ae4849a0ffc05020300c0d
I20260810 07:28:09.783144 5267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:28:09.792282 5267 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:28:09.793679 5267 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:28:09.802821 5267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7: Generated new TSK 0
I20260810 07:28:09.803714 5267 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
I20260810 07:28:09.841625 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:0
--local_ip_for_outbound_sockets=127.12.157.193
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:43331
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:34025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:09.960090 5289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:09.960366 5289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:09.960417 5289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:09.964378 5289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:09.964447 5289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:09.964507 5289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:28:09.969818 5289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:34025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:43331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:28:09.971021 5289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:09.972069 5289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:09.982251 5299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:09.982251 5305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:09.982266 5300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:09.982570 5289 server_base.cc:1034] running on GCE node
I20260810 07:28:09.983080 5289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:09.983708 5289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:09.984908 5289 hybrid_clock.cc:648] HybridClock initialized: now 1786346889984892 us; error 27 us; skew 500 ppm
Aug 10 07:28:09 dist-test-slave-ffrd krb5kdc[5100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346889, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:09.988013 5289 init.cc:377] Logged in from keytab as kudu/127.12.157.193@KRBTEST.COM (short username kudu)
I20260810 07:28:09.989013 5289 webserver.cc:460] Webserver started at http://127.12.157.193:40987/ using document root <none> and password file <none>
I20260810 07:28:09.989600 5289 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:09.989694 5289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:09.989935 5289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:09.991994 5289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/data/instance:
uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9"
format_stamp: "Formatted at 2026-08-10 07:28:09 on dist-test-slave-ffrd"
server_key: "b4cb964d9b52f43ddb29c25bb7af60df"
server_key_iv: "33514581e8f8f922a92748b9c791f8ee"
server_key_version: "encryptionkey@0"
I20260810 07:28:09.992612 5289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance:
uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9"
format_stamp: "Formatted at 2026-08-10 07:28:09 on dist-test-slave-ffrd"
server_key: "b4cb964d9b52f43ddb29c25bb7af60df"
server_key_iv: "33514581e8f8f922a92748b9c791f8ee"
server_key_version: "encryptionkey@0"
I20260810 07:28:09.996279 5289 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 07:28:09.998759 5316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:10.000046 5289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 07:28:10.000197 5289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9"
format_stamp: "Formatted at 2026-08-10 07:28:09 on dist-test-slave-ffrd"
server_key: "b4cb964d9b52f43ddb29c25bb7af60df"
server_key_iv: "33514581e8f8f922a92748b9c791f8ee"
server_key_version: "encryptionkey@0"
I20260810 07:28:10.000327 5289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:10.030157 5289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:10.033480 5289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:10.033699 5289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:10.034337 5289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:10.035481 5289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:28:10.035573 5289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:10.035669 5289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:28:10.035720 5289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:10.046034 5289 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:46095
I20260810 07:28:10.046057 5503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:46095 every 8 connection(s)
I20260810 07:28:10.047055 5289 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:28:10.049455 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 5289
I20260810 07:28:10.049561 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance
I20260810 07:28:10.049805 12919 external_mini_cluster.cc:1428] Setting key 9ee1bc67b178de17f103e8719d854af5
Aug 10 07:28:10 dist-test-slave-ffrd krb5kdc[5100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346889, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.157.194@KRBTEST.COM; defaulting to no policy
I20260810 07:28:10.064203 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:10.049284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:53185 (local address 127.12.157.254:43331)
0810 07:28:10.049474 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:10.049480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:10.050393 (+ 913us) server_negotiation.cc:407] Connection header received
0810 07:28:10.052155 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:10.052158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:10.052204 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:10.052285 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:10.054067 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.054978 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.056219 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.056661 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.056820 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:10.059945 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:10.059952 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:10.059956 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:10.059991 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:10.061814 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:10.062412 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:10.062415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:10.062417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:10.062498 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:10.062917 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:10.062920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:10.062921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:10.063127 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:10.063241 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:10.063871 (+ 630us) server_negotiation.cc:299] Negotiation successful
0810 07:28:10.064027 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.194@KRBTEST.COM" created.
I20260810 07:28:10.065317 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:10.049677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.193:53185)
0810 07:28:10.050208 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:10.050267 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:10.051889 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:10.052498 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:10.052511 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:10.053117 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:28:10.053856 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:10.053881 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.055138 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.055141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:10.056072 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:10.056082 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.056865 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.056869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:10.057127 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:10.057816 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:28:10.057835 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:28:10.059728 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:28:10.061984 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:10.061993 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:10.062014 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:10.062288 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:10.062590 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:10.062593 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:10.062594 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:10.062781 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:10.063240 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:28:10.063249 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:28:10.063537 (+ 288us) client_negotiation.cc:769] Sending connection context
0810 07:28:10.063737 (+ 200us) client_negotiation.cc:240] Negotiation successful
0810 07:28:10.064103 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":132,"threads_started":1}
I20260810 07:28:10.066716 5504 heartbeater.cc:347] Connected to a master server at 127.12.157.254:43331
I20260810 07:28:10.067018 5504 heartbeater.cc:464] Registering TS with master...
I20260810 07:28:10.067678 5504 heartbeater.cc:511] Master 127.12.157.254:43331 requested a full tablet report, sending...
I20260810 07:28:10.069492 5183 ts_manager.cc:194] Registered new tserver with Master: 294a3cf7eacc474dbec76b0bcd8fa9f9 (127.12.157.193:46095)
I20260810 07:28:10.070640 5183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.193@KRBTEST.COM'} at 127.12.157.193:53185
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
I20260810 07:28:10.083931 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:0
--local_ip_for_outbound_sockets=127.12.157.194
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:43331
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:34025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:10.205165 5511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:10.205432 5511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:10.205478 5511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:10.209559 5511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:10.209621 5511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:10.209694 5511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:28:10.215294 5511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:34025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:43331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:28:10.216405 5511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:10.217445 5511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:10.228034 5517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:10.228143 5511 server_base.cc:1034] running on GCE node
W20260810 07:28:10.228036 5516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:10.228034 5521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:10.228791 5511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:10.229574 5511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:10.230780 5511 hybrid_clock.cc:648] HybridClock initialized: now 1786346890230759 us; error 33 us; skew 500 ppm
Aug 10 07:28:10 dist-test-slave-ffrd krb5kdc[5100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346890, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:10.234071 5511 init.cc:377] Logged in from keytab as kudu/127.12.157.194@KRBTEST.COM (short username kudu)
I20260810 07:28:10.235040 5511 webserver.cc:460] Webserver started at http://127.12.157.194:34893/ using document root <none> and password file <none>
I20260810 07:28:10.235646 5511 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:10.235745 5511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:10.236033 5511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:10.238303 5511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/data/instance:
uuid: "20d0364e3e7845678f58e593357e654c"
format_stamp: "Formatted at 2026-08-10 07:28:10 on dist-test-slave-ffrd"
server_key: "0359780f707658a68d0e03835af0aca9"
server_key_iv: "90b48d5108c86a2862bba1d71b3af763"
server_key_version: "encryptionkey@0"
I20260810 07:28:10.238919 5511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance:
uuid: "20d0364e3e7845678f58e593357e654c"
format_stamp: "Formatted at 2026-08-10 07:28:10 on dist-test-slave-ffrd"
server_key: "0359780f707658a68d0e03835af0aca9"
server_key_iv: "90b48d5108c86a2862bba1d71b3af763"
server_key_version: "encryptionkey@0"
I20260810 07:28:10.243712 5511 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260810 07:28:10.246424 5535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:10.247733 5511 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 07:28:10.247893 5511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "20d0364e3e7845678f58e593357e654c"
format_stamp: "Formatted at 2026-08-10 07:28:10 on dist-test-slave-ffrd"
server_key: "0359780f707658a68d0e03835af0aca9"
server_key_iv: "90b48d5108c86a2862bba1d71b3af763"
server_key_version: "encryptionkey@0"
I20260810 07:28:10.248023 5511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:10.274065 5511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:10.277401 5511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:10.277618 5511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:10.278285 5511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:10.279145 5511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:28:10.279222 5511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:10.279299 5511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:28:10.279350 5511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:10.289062 5511 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:38525
I20260810 07:28:10.289095 5711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:38525 every 8 connection(s)
I20260810 07:28:10.290002 5511 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:28:10.292519 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 5511
I20260810 07:28:10.292644 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance
I20260810 07:28:10.293082 12919 external_mini_cluster.cc:1428] Setting key 297352255a5c728ca72429a970da8683
Aug 10 07:28:10 dist-test-slave-ffrd krb5kdc[5100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346890, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.157.195@KRBTEST.COM; defaulting to no policy
I20260810 07:28:10.306654 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:10.291996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:48205 (local address 127.12.157.254:43331)
0810 07:28:10.292153 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:10.292157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:10.293245 (+ 1088us) server_negotiation.cc:407] Connection header received
0810 07:28:10.294839 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:10.294843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:10.294902 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:10.295003 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:10.296203 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.296762 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.298123 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.298382 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.298472 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:10.302325 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:10.302329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:10.302331 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:10.302478 (+ 147us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:10.304280 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:10.305004 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:10.305008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:10.305010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:10.305097 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:10.305479 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:10.305483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:10.305484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:10.305689 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:10.305798 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:10.306346 (+ 548us) server_negotiation.cc:299] Negotiation successful
0810 07:28:10.306483 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.195@KRBTEST.COM" created.
I20260810 07:28:10.307538 5718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:10.292432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.194:48205)
0810 07:28:10.293047 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:10.293109 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:10.294622 (+ 1513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:10.295181 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:10.295188 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:10.295550 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:28:10.296052 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:10.296066 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.296910 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.296914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:10.297985 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:10.297999 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.298548 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.298555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:10.298799 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:10.299531 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:28:10.299558 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:28:10.302119 (+ 2561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:28:10.304514 (+ 2395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:10.304525 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:10.304544 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:10.304846 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:10.305198 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:10.305202 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:10.305204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:10.305369 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:10.305799 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:28:10.305807 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:28:10.306090 (+ 283us) client_negotiation.cc:769] Sending connection context
0810 07:28:10.306286 (+ 196us) client_negotiation.cc:240] Negotiation successful
0810 07:28:10.306652 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":186,"threads_started":1}
I20260810 07:28:10.308871 5715 heartbeater.cc:347] Connected to a master server at 127.12.157.254:43331
I20260810 07:28:10.309170 5715 heartbeater.cc:464] Registering TS with master...
I20260810 07:28:10.309701 5715 heartbeater.cc:511] Master 127.12.157.254:43331 requested a full tablet report, sending...
I20260810 07:28:10.311049 5183 ts_manager.cc:194] Registered new tserver with Master: 20d0364e3e7845678f58e593357e654c (127.12.157.194:38525)
I20260810 07:28:10.311745 5183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.194@KRBTEST.COM'} at 127.12.157.194:48205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
I20260810 07:28:10.327886 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:0
--local_ip_for_outbound_sockets=127.12.157.195
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:43331
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:34025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:10.447242 5723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:10.447506 5723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:10.447552 5723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:10.451689 5723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:10.451756 5723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:10.451974 5723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:28:10.457531 5723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:34025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:43331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:28:10.458598 5723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:10.459620 5723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:10.469545 5733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:10.469561 5736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:10.469616 5731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:10.469849 5723 server_base.cc:1034] running on GCE node
I20260810 07:28:10.470288 5723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:10.470930 5723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:10.472095 5723 hybrid_clock.cc:648] HybridClock initialized: now 1786346890472076 us; error 31 us; skew 500 ppm
Aug 10 07:28:10 dist-test-slave-ffrd krb5kdc[5100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346890, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:10.475176 5723 init.cc:377] Logged in from keytab as kudu/127.12.157.195@KRBTEST.COM (short username kudu)
I20260810 07:28:10.476203 5723 webserver.cc:460] Webserver started at http://127.12.157.195:34209/ using document root <none> and password file <none>
I20260810 07:28:10.477011 5723 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:10.477084 5723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:10.477260 5723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:28:10.479285 5723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/data/instance:
uuid: "553eefb7d44940aa9c80f507424b214f"
format_stamp: "Formatted at 2026-08-10 07:28:10 on dist-test-slave-ffrd"
server_key: "720466e9669ae0f8459b5001e5b0878b"
server_key_iv: "dc2b88e2319dbdda653d426e6c3a14df"
server_key_version: "encryptionkey@0"
I20260810 07:28:10.479807 5723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance:
uuid: "553eefb7d44940aa9c80f507424b214f"
format_stamp: "Formatted at 2026-08-10 07:28:10 on dist-test-slave-ffrd"
server_key: "720466e9669ae0f8459b5001e5b0878b"
server_key_iv: "dc2b88e2319dbdda653d426e6c3a14df"
server_key_version: "encryptionkey@0"
I20260810 07:28:10.483996 5723 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 07:28:10.486469 5744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:10.487721 5723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 07:28:10.487823 5723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "553eefb7d44940aa9c80f507424b214f"
format_stamp: "Formatted at 2026-08-10 07:28:10 on dist-test-slave-ffrd"
server_key: "720466e9669ae0f8459b5001e5b0878b"
server_key_iv: "dc2b88e2319dbdda653d426e6c3a14df"
server_key_version: "encryptionkey@0"
I20260810 07:28:10.487941 5723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:10.512121 5723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:10.515516 5723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:10.515690 5723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:10.516530 5723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:10.517472 5723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:28:10.517525 5723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:10.517568 5723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:28:10.517584 5723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:10.527633 5723 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:41341
I20260810 07:28:10.527657 5942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:41341 every 8 connection(s)
I20260810 07:28:10.528688 5723 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:28:10.536474 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 5723
I20260810 07:28:10.536623 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance
I20260810 07:28:10.536918 12919 external_mini_cluster.cc:1428] Setting key 582e4cc34cb0cad26fb17a2bcf9aada1
Aug 10 07:28:10 dist-test-slave-ffrd krb5kdc[5100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346890, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:28:10.544708 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:10.531096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:54867 (local address 127.12.157.254:43331)
0810 07:28:10.531326 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:10.531331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:10.532778 (+ 1447us) server_negotiation.cc:407] Connection header received
0810 07:28:10.534524 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:10.534528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:10.534579 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:10.534660 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:10.535816 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.536503 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.538026 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.538262 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.538380 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:10.541055 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:10.541059 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:10.541061 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:10.541096 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:10.542535 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:10.543096 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:10.543099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:10.543101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:10.543189 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:10.543560 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:10.543563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:10.543565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:10.543774 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:10.543883 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:10.544415 (+ 532us) server_negotiation.cc:299] Negotiation successful
0810 07:28:10.544537 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260810 07:28:10.545593 5946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:10.531341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.195:54867)
0810 07:28:10.532563 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:10.532630 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:10.534312 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:10.534802 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:10.534809 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:10.535168 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:28:10.535655 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:10.535668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.536676 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.536682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:10.537899 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:10.537911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.538439 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.538443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:10.538627 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:10.539296 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:28:10.539313 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:28:10.540849 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:28:10.542673 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:10.542679 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:10.542695 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:10.542979 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:10.543299 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:10.543302 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:10.543304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:10.543452 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:10.543898 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:28:10.543907 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:28:10.544160 (+ 253us) client_negotiation.cc:769] Sending connection context
0810 07:28:10.544356 (+ 196us) client_negotiation.cc:240] Negotiation successful
0810 07:28:10.544691 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"thread_start_us":179,"threads_started":1}
I20260810 07:28:10.546913 5943 heartbeater.cc:347] Connected to a master server at 127.12.157.254:43331
I20260810 07:28:10.547153 5943 heartbeater.cc:464] Registering TS with master...
I20260810 07:28:10.547646 5943 heartbeater.cc:511] Master 127.12.157.254:43331 requested a full tablet report, sending...
I20260810 07:28:10.549064 5183 ts_manager.cc:194] Registered new tserver with Master: 553eefb7d44940aa9c80f507424b214f (127.12.157.195:41341)
I20260810 07:28:10.549749 5183 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.195@KRBTEST.COM'} at 127.12.157.195:54867
I20260810 07:28:10.551033 12919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:28:10.561368 5250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:10.553375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46962 (local address 127.12.157.254:43331)
0810 07:28:10.553531 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:10.553536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:10.553799 (+ 263us) server_negotiation.cc:407] Connection header received
0810 07:28:10.553907 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:10.553910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:10.553949 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:10.554022 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:10.554714 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.555234 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.556075 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.556289 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.556388 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:10.557815 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:10.557819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:10.557821 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:10.557846 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:10.559599 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:10.560061 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:10.560063 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:10.560065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:10.560142 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:10.560472 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:10.560475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:10.560476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:10.560697 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:10.560808 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:10.561095 (+ 287us) server_negotiation.cc:299] Negotiation successful
0810 07:28:10.561193 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 07:28:10.564711 5183 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46962:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:28:10.567857 5183 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:28:10.582758 5956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:10.577182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:46095 (local address 127.0.0.1:44338)
0810 07:28:10.577589 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:10.577642 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:10.577768 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:10.578339 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:10.578347 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:10.578379 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:10.578555 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:10.578564 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.580075 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.580080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:10.581087 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:10.581096 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.582204 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.582208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:10.582343 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:10.582392 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 07:28:10.582485 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 07:28:10.582603 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":94,"threads_started":1}
I20260810 07:28:10.583675 5957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:10.577369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44338 (local address 127.12.157.193:46095)
0810 07:28:10.577727 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:10.577733 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:10.577760 (+ 27us) server_negotiation.cc:407] Connection header received
0810 07:28:10.577919 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:10.577924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:10.578145 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:10.578737 (+ 592us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:10.578891 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.579916 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.581224 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.582066 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.583180 (+ 1114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:10.583343 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:10.583399 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 07:28:10.583499 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":133,"threads_started":1}
I20260810 07:28:10.583796 5965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:10.578306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:41341 (local address 127.0.0.1:59072)
0810 07:28:10.578531 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:10.578545 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:10.578622 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:10.579764 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:10.579767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:10.579786 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:10.579893 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:10.579900 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.581244 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.581247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:10.582424 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:10.582432 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.583404 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.583407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:10.583524 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:10.583553 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 07:28:10.583595 (+ 42us) client_negotiation.cc:240] Negotiation successful
0810 07:28:10.583648 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":2,"thread_start_us":96,"threads_started":1}
I20260810 07:28:10.583844 5961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:10.578124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44144 (local address 127.12.157.194:38525)
0810 07:28:10.578617 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:10.578629 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:10.578653 (+ 24us) server_negotiation.cc:407] Connection header received
0810 07:28:10.578744 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:10.578750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:10.578975 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:10.579455 (+ 480us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:10.579625 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.580779 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.582354 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.583175 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.583296 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:10.583412 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:10.583590 (+ 178us) server_negotiation.cc:299] Negotiation successful
0810 07:28:10.583684 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":171,"threads_started":1}
I20260810 07:28:10.584180 5958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:10.577959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:38525 (local address 127.0.0.1:44144)
0810 07:28:10.578203 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:10.578218 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:10.578330 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:10.579148 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:10.579153 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:10.579173 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:10.579290 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:10.579297 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.580973 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.580977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:10.582237 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:10.582248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.583307 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.583313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:10.583447 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:10.583473 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 07:28:10.583548 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 07:28:10.583626 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":22,"thread_start_us":113,"threads_started":1}
I20260810 07:28:10.584537 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:10.578836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59072 (local address 127.12.157.195:41341)
0810 07:28:10.579307 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:10.579337 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:10.579361 (+ 24us) server_negotiation.cc:407] Connection header received
0810 07:28:10.579438 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:10.579443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:10.579612 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:10.579762 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:10.580017 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.581125 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.582544 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.583261 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.584082 (+ 821us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:10.584210 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:10.584263 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 07:28:10.584382 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":158,"threads_started":1}
I20260810 07:28:10.586539 5396 tablet_service.cc:1467] Processing CreateTablet for tablet 09c82761a3f7466eaed6fdf871f462c2 (DEFAULT_TABLE table=test-table [id=a7a5d1c8e3b74c148c71cde5874398ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:28:10.586892 5838 tablet_service.cc:1467] Processing CreateTablet for tablet 09c82761a3f7466eaed6fdf871f462c2 (DEFAULT_TABLE table=test-table [id=a7a5d1c8e3b74c148c71cde5874398ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:28:10.586892 5611 tablet_service.cc:1467] Processing CreateTablet for tablet 09c82761a3f7466eaed6fdf871f462c2 (DEFAULT_TABLE table=test-table [id=a7a5d1c8e3b74c148c71cde5874398ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:28:10.587481 5396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09c82761a3f7466eaed6fdf871f462c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:10.587898 5838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09c82761a3f7466eaed6fdf871f462c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:10.587890 5611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09c82761a3f7466eaed6fdf871f462c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:10.594429 5967 tablet_bootstrap.cc:492] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9: Bootstrap starting.
I20260810 07:28:10.595427 5973 tablet_bootstrap.cc:492] T 09c82761a3f7466eaed6fdf871f462c2 P 20d0364e3e7845678f58e593357e654c: Bootstrap starting.
I20260810 07:28:10.596619 5967 tablet_bootstrap.cc:654] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:10.597311 5970 tablet_bootstrap.cc:492] T 09c82761a3f7466eaed6fdf871f462c2 P 553eefb7d44940aa9c80f507424b214f: Bootstrap starting.
I20260810 07:28:10.597523 5967 log.cc:824] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:10.597716 5973 tablet_bootstrap.cc:654] T 09c82761a3f7466eaed6fdf871f462c2 P 20d0364e3e7845678f58e593357e654c: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:10.598542 5973 log.cc:824] T 09c82761a3f7466eaed6fdf871f462c2 P 20d0364e3e7845678f58e593357e654c: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:10.599382 5970 tablet_bootstrap.cc:654] T 09c82761a3f7466eaed6fdf871f462c2 P 553eefb7d44940aa9c80f507424b214f: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:10.599748 5967 tablet_bootstrap.cc:492] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9: No bootstrap required, opened a new log
I20260810 07:28:10.599954 5967 ts_tablet_manager.cc:1397] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260810 07:28:10.600564 5973 tablet_bootstrap.cc:492] T 09c82761a3f7466eaed6fdf871f462c2 P 20d0364e3e7845678f58e593357e654c: No bootstrap required, opened a new log
I20260810 07:28:10.600622 5970 log.cc:824] T 09c82761a3f7466eaed6fdf871f462c2 P 553eefb7d44940aa9c80f507424b214f: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:10.600744 5973 ts_tablet_manager.cc:1397] T 09c82761a3f7466eaed6fdf871f462c2 P 20d0364e3e7845678f58e593357e654c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 07:28:10.602650 5970 tablet_bootstrap.cc:492] T 09c82761a3f7466eaed6fdf871f462c2 P 553eefb7d44940aa9c80f507424b214f: No bootstrap required, opened a new log
I20260810 07:28:10.602843 5970 ts_tablet_manager.cc:1397] T 09c82761a3f7466eaed6fdf871f462c2 P 553eefb7d44940aa9c80f507424b214f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260810 07:28:10.604038 5967 raft_consensus.cc:348] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46095 } } peers { permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 } } peers { permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 } }
I20260810 07:28:10.604328 5967 raft_consensus.cc:374] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:10.604395 5967 raft_consensus.cc:729] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294a3cf7eacc474dbec76b0bcd8fa9f9, State: Initialized, Role: FOLLOWER
I20260810 07:28:10.604482 5973 raft_consensus.cc:348] T 09c82761a3f7466eaed6fdf871f462c2 P 20d0364e3e7845678f58e593357e654c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46095 } } peers { permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 } } peers { permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 } }
I20260810 07:28:10.604717 5973 raft_consensus.cc:374] T 09c82761a3f7466eaed6fdf871f462c2 P 20d0364e3e7845678f58e593357e654c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:10.604775 5973 raft_consensus.cc:729] T 09c82761a3f7466eaed6fdf871f462c2 P 20d0364e3e7845678f58e593357e654c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20d0364e3e7845678f58e593357e654c, State: Initialized, Role: FOLLOWER
I20260810 07:28:10.604970 5967 consensus_queue.cc:260] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46095 } } peers { permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 } } peers { permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 } }
I20260810 07:28:10.605216 5973 consensus_queue.cc:260] T 09c82761a3f7466eaed6fdf871f462c2 P 20d0364e3e7845678f58e593357e654c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46095 } } peers { permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 } } peers { permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 } }
I20260810 07:28:10.605719 5504 heartbeater.cc:503] Master 127.12.157.254:43331 was elected leader, sending a full tablet report...
I20260810 07:28:10.606031 5967 ts_tablet_manager.cc:1428] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 07:28:10.606129 5973 ts_tablet_manager.cc:1428] T 09c82761a3f7466eaed6fdf871f462c2 P 20d0364e3e7845678f58e593357e654c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 07:28:10.606453 5715 heartbeater.cc:503] Master 127.12.157.254:43331 was elected leader, sending a full tablet report...
I20260810 07:28:10.606954 5970 raft_consensus.cc:348] T 09c82761a3f7466eaed6fdf871f462c2 P 553eefb7d44940aa9c80f507424b214f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46095 } } peers { permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 } } peers { permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 } }
I20260810 07:28:10.607240 5970 raft_consensus.cc:374] T 09c82761a3f7466eaed6fdf871f462c2 P 553eefb7d44940aa9c80f507424b214f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:10.607299 5970 raft_consensus.cc:729] T 09c82761a3f7466eaed6fdf871f462c2 P 553eefb7d44940aa9c80f507424b214f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 553eefb7d44940aa9c80f507424b214f, State: Initialized, Role: FOLLOWER
I20260810 07:28:10.607702 5970 consensus_queue.cc:260] T 09c82761a3f7466eaed6fdf871f462c2 P 553eefb7d44940aa9c80f507424b214f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46095 } } peers { permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 } } peers { permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 } }
I20260810 07:28:10.608357 5943 heartbeater.cc:503] Master 127.12.157.254:43331 was elected leader, sending a full tablet report...
I20260810 07:28:10.608728 5970 ts_tablet_manager.cc:1428] T 09c82761a3f7466eaed6fdf871f462c2 P 553eefb7d44940aa9c80f507424b214f: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
W20260810 07:28:10.780020 5945 tablet.cc:2369] T 09c82761a3f7466eaed6fdf871f462c2 P 553eefb7d44940aa9c80f507424b214f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 07:28:10.791210 5717 tablet.cc:2369] T 09c82761a3f7466eaed6fdf871f462c2 P 20d0364e3e7845678f58e593357e654c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 07:28:10.798533 5507 tablet.cc:2369] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:28:10.835850 5979 raft_consensus.cc:482] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:28:10.836143 5979 raft_consensus.cc:504] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46095 } } peers { permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 } } peers { permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 } }
I20260810 07:28:10.837447 5979 leader_election.cc:288] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20d0364e3e7845678f58e593357e654c (127.12.157.194:38525), 553eefb7d44940aa9c80f507424b214f (127.12.157.195:41341)
I20260810 07:28:10.842052 5961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:10.837919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:52191 (local address 127.12.157.194:38525)
0810 07:28:10.838107 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:10.838111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:10.838331 (+ 220us) server_negotiation.cc:407] Connection header received
0810 07:28:10.838469 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:10.838472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:10.838540 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:10.838641 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:10.839032 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.839653 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.841046 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.841457 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.841551 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:10.841635 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:10.841839 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 07:28:10.841923 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 07:28:10.842064 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:10.837847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:38525 (local address 127.12.157.193:52191)
0810 07:28:10.838173 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:10.838190 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:10.838332 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:10.838675 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:10.838678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:10.838702 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:10.838902 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:10.838910 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.839767 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.839770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:10.840852 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:10.840863 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.841584 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.841591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:10.841687 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:10.841719 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 07:28:10.841793 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 07:28:10.841849 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":76,"spinlock_wait_cycles":896,"thread_start_us":132,"threads_started":1}
I20260810 07:28:10.842121 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:10.838582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:33053 (local address 127.12.157.195:41341)
0810 07:28:10.838738 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:10.838743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:10.838757 (+ 14us) server_negotiation.cc:407] Connection header received
0810 07:28:10.838850 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:10.838855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:10.838902 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:10.838988 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:10.839290 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.839837 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.841065 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.841496 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.841637 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:10.841693 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:10.841919 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 07:28:10.841996 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 07:28:10.842077 5988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:10.838294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:41341 (local address 127.12.157.193:33053)
0810 07:28:10.838647 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:10.838661 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:10.838750 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:10.839025 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:10.839028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:10.839047 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:10.839159 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:10.839164 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.840020 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.840023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:10.840882 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:10.840892 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.841656 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.841661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:10.841753 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:10.841792 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 07:28:10.841882 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 07:28:10.841945 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":21,"thread_start_us":107,"threads_started":1}
I20260810 07:28:10.843009 5643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09c82761a3f7466eaed6fdf871f462c2" candidate_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20d0364e3e7845678f58e593357e654c" is_pre_election: true
I20260810 07:28:10.843024 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09c82761a3f7466eaed6fdf871f462c2" candidate_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "553eefb7d44940aa9c80f507424b214f" is_pre_election: true
I20260810 07:28:10.843338 5866 raft_consensus.cc:2454] T 09c82761a3f7466eaed6fdf871f462c2 P 553eefb7d44940aa9c80f507424b214f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294a3cf7eacc474dbec76b0bcd8fa9f9 in term 0.
I20260810 07:28:10.843339 5643 raft_consensus.cc:2454] T 09c82761a3f7466eaed6fdf871f462c2 P 20d0364e3e7845678f58e593357e654c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294a3cf7eacc474dbec76b0bcd8fa9f9 in term 0.
I20260810 07:28:10.843875 5322 leader_election.cc:302] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20d0364e3e7845678f58e593357e654c, 294a3cf7eacc474dbec76b0bcd8fa9f9; no voters:
I20260810 07:28:10.844194 5979 raft_consensus.cc:2781] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:28:10.844321 5979 raft_consensus.cc:482] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:28:10.844431 5979 raft_consensus.cc:3037] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:10.845715 5979 raft_consensus.cc:504] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46095 } } peers { permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 } } peers { permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 } }
I20260810 07:28:10.846076 5979 leader_election.cc:288] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [CANDIDATE]: Term 1 election: Requested vote from peers 20d0364e3e7845678f58e593357e654c (127.12.157.194:38525), 553eefb7d44940aa9c80f507424b214f (127.12.157.195:41341)
I20260810 07:28:10.846459 5643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09c82761a3f7466eaed6fdf871f462c2" candidate_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20d0364e3e7845678f58e593357e654c"
I20260810 07:28:10.846577 5866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09c82761a3f7466eaed6fdf871f462c2" candidate_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "553eefb7d44940aa9c80f507424b214f"
I20260810 07:28:10.846577 5643 raft_consensus.cc:3037] T 09c82761a3f7466eaed6fdf871f462c2 P 20d0364e3e7845678f58e593357e654c [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:10.846731 5866 raft_consensus.cc:3037] T 09c82761a3f7466eaed6fdf871f462c2 P 553eefb7d44940aa9c80f507424b214f [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:10.847690 5643 raft_consensus.cc:2454] T 09c82761a3f7466eaed6fdf871f462c2 P 20d0364e3e7845678f58e593357e654c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294a3cf7eacc474dbec76b0bcd8fa9f9 in term 1.
I20260810 07:28:10.847800 5866 raft_consensus.cc:2454] T 09c82761a3f7466eaed6fdf871f462c2 P 553eefb7d44940aa9c80f507424b214f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294a3cf7eacc474dbec76b0bcd8fa9f9 in term 1.
I20260810 07:28:10.848013 5322 leader_election.cc:302] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20d0364e3e7845678f58e593357e654c, 294a3cf7eacc474dbec76b0bcd8fa9f9; no voters:
I20260810 07:28:10.848208 5979 raft_consensus.cc:2781] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:28:10.848448 5979 raft_consensus.cc:686] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 1 LEADER]: Becoming Leader. State: Replica: 294a3cf7eacc474dbec76b0bcd8fa9f9, State: Running, Role: LEADER
I20260810 07:28:10.848802 5979 consensus_queue.cc:237] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46095 } } peers { permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 } } peers { permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 } }
I20260810 07:28:10.852483 5183 catalog_manager.cc:5495] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 294a3cf7eacc474dbec76b0bcd8fa9f9 (127.12.157.193). New cstate: current_term: 1 leader_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 } health_report { overall_health: UNKNOWN } } }
I20260810 07:28:10.882400 5957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:10.878291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44342 (local address 127.12.157.193:46095)
0810 07:28:10.878728 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:10.878733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:10.878747 (+ 14us) server_negotiation.cc:407] Connection header received
0810 07:28:10.878830 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:10.878833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:10.878892 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:10.878987 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:28:10.879253 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.879778 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.880716 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:10.881041 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:10.881136 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:10.881470 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:28:10.881930 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:28:10.882029 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:10.882164 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 07:28:10.882226 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349}
I20260810 07:28:10.888593 5866 raft_consensus.cc:1261] T 09c82761a3f7466eaed6fdf871f462c2 P 553eefb7d44940aa9c80f507424b214f [term 1 FOLLOWER]: Refusing update from remote peer 294a3cf7eacc474dbec76b0bcd8fa9f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:28:10.888623 5643 raft_consensus.cc:1261] T 09c82761a3f7466eaed6fdf871f462c2 P 20d0364e3e7845678f58e593357e654c [term 1 FOLLOWER]: Refusing update from remote peer 294a3cf7eacc474dbec76b0bcd8fa9f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:28:10.889415 5979 consensus_queue.cc:1035] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:28:10.889674 5990 consensus_queue.cc:1035] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:28:10.896490 5994 mvcc.cc:205] Tried to move back new op lower bound from 7316876865071878144 to 7316876864919592960. Current Snapshot: MvccSnapshot[applied={T|T < 7316876865071878144}]
I20260810 07:28:10.899354 5997 mvcc.cc:205] Tried to move back new op lower bound from 7316876865071878144 to 7316876864919592960. Current Snapshot: MvccSnapshot[applied={T|T < 7316876865071878144}]
I20260810 07:28:10.899510 5995 mvcc.cc:205] Tried to move back new op lower bound from 7316876865071878144 to 7316876864919592960. Current Snapshot: MvccSnapshot[applied={T|T < 7316876865071878144}]
I20260810 07:28:10.906356 5183 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:28:10.906548 5183 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:28:10.909126 5183 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7: Sending DeleteTablet for 3 replicas of tablet 09c82761a3f7466eaed6fdf871f462c2
I20260810 07:28:10.910111 5611 tablet_service.cc:1514] Processing DeleteTablet for tablet 09c82761a3f7466eaed6fdf871f462c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:28:10 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:44144
I20260810 07:28:10.910111 5396 tablet_service.cc:1514] Processing DeleteTablet for tablet 09c82761a3f7466eaed6fdf871f462c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:28:10 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:44338
I20260810 07:28:10.910218 5838 tablet_service.cc:1514] Processing DeleteTablet for tablet 09c82761a3f7466eaed6fdf871f462c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:28:10 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:59072
I20260810 07:28:10.910778 6014 tablet_replica.cc:330] T 09c82761a3f7466eaed6fdf871f462c2 P 553eefb7d44940aa9c80f507424b214f: stopping tablet replica
I20260810 07:28:10.910970 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 5289
I20260810 07:28:10.933579 6014 raft_consensus.cc:2229] T 09c82761a3f7466eaed6fdf871f462c2 P 553eefb7d44940aa9c80f507424b214f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:28:10.934067 6014 raft_consensus.cc:2258] T 09c82761a3f7466eaed6fdf871f462c2 P 553eefb7d44940aa9c80f507424b214f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:28:10.935468 6013 tablet_replica.cc:330] T 09c82761a3f7466eaed6fdf871f462c2 P 20d0364e3e7845678f58e593357e654c: stopping tablet replica
I20260810 07:28:10.936570 6013 raft_consensus.cc:2229] T 09c82761a3f7466eaed6fdf871f462c2 P 20d0364e3e7845678f58e593357e654c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:28:10.936980 6013 raft_consensus.cc:2258] T 09c82761a3f7466eaed6fdf871f462c2 P 20d0364e3e7845678f58e593357e654c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:28:10.938123 6014 ts_tablet_manager.cc:1905] T 09c82761a3f7466eaed6fdf871f462c2 P 553eefb7d44940aa9c80f507424b214f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:28:10.938611 6013 ts_tablet_manager.cc:1905] T 09c82761a3f7466eaed6fdf871f462c2 P 20d0364e3e7845678f58e593357e654c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:28:10.941012 6013 ts_tablet_manager.cc:1918] T 09c82761a3f7466eaed6fdf871f462c2 P 20d0364e3e7845678f58e593357e654c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:28:10.941149 6013 log.cc:1196] T 09c82761a3f7466eaed6fdf871f462c2 P 20d0364e3e7845678f58e593357e654c: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/wal/wals/09c82761a3f7466eaed6fdf871f462c2
I20260810 07:28:10.941643 6013 ts_tablet_manager.cc:1939] T 09c82761a3f7466eaed6fdf871f462c2 P 20d0364e3e7845678f58e593357e654c: Deleting consensus metadata
I20260810 07:28:10.942040 6014 ts_tablet_manager.cc:1918] T 09c82761a3f7466eaed6fdf871f462c2 P 553eefb7d44940aa9c80f507424b214f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:28:10.942164 6014 log.cc:1196] T 09c82761a3f7466eaed6fdf871f462c2 P 553eefb7d44940aa9c80f507424b214f: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/wal/wals/09c82761a3f7466eaed6fdf871f462c2
I20260810 07:28:10.942515 6014 ts_tablet_manager.cc:1939] T 09c82761a3f7466eaed6fdf871f462c2 P 553eefb7d44940aa9c80f507424b214f: Deleting consensus metadata
I20260810 07:28:10.943192 5158 catalog_manager.cc:4841] TS 20d0364e3e7845678f58e593357e654c (127.12.157.194:38525): tablet 09c82761a3f7466eaed6fdf871f462c2 (table test-table [id=a7a5d1c8e3b74c148c71cde5874398ec]) successfully deleted
I20260810 07:28:10.943881 5158 catalog_manager.cc:4841] TS 553eefb7d44940aa9c80f507424b214f (127.12.157.195:41341): tablet 09c82761a3f7466eaed6fdf871f462c2 (table test-table [id=a7a5d1c8e3b74c148c71cde5874398ec]) successfully deleted
W20260810 07:28:10.946203 5158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.157.193:46095 (error 108)
I20260810 07:28:10.946566 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 5511
I20260810 07:28:10.950184 5958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:10.949846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:46095 (local address 127.0.0.1:44354)
0810 07:28:10.949992 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:10.950078 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.193:46095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260810 07:28:10.950490 5158 catalog_manager.cc:4584] TS 294a3cf7eacc474dbec76b0bcd8fa9f9 (127.12.157.193:46095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 09c82761a3f7466eaed6fdf871f462c2: Network error: Client connection negotiation failed: client connection to 127.12.157.193:46095: connect: Connection refused (error 111)
I20260810 07:28:10.961150 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 5723
I20260810 07:28:10.976086 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 5125
I20260810 07:28:30.999835 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:43331
--webserver_interface=127.12.157.254
--webserver_port=42839
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:34025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:43331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:31.116623 6016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:31.116894 6016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:31.116935 6016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:31.120872 6016 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 07:28:31.120923 6016 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 07:28:31.120954 6016 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 07:28:31.120970 6016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:28:31.120986 6016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:31.121004 6016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:31.121021 6016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:28:31.121035 6016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:28:31.126269 6016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:34025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:43331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:43331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=42839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:28:31.127231 6016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:31.128278 6016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:31.143296 6027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:31.143463 6030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:31.143594 6016 server_base.cc:1034] running on GCE node
W20260810 07:28:31.143690 6025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:31.144393 6016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:31.145879 6016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:31.147102 6016 hybrid_clock.cc:648] HybridClock initialized: now 1786346911147091 us; error 31 us; skew 500 ppm
Aug 10 07:28:31 dist-test-slave-ffrd krb5kdc[5100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346911, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:31.150223 6016 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:28:31.151268 6016 webserver.cc:460] Webserver started at http://127.12.157.254:42839/ using document root <none> and password file <none>
I20260810 07:28:31.151852 6016 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:31.151928 6016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:31.156633 6016 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:28:31.159242 6046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:31.160620 6016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 07:28:31.160745 6016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "3bb300bc5b3c41a1811369fec9238ce7"
format_stamp: "Formatted at 2026-08-10 07:28:09 on dist-test-slave-ffrd"
server_key: "cd65317d4fb8261f01ed216c32c2f3f5"
server_key_iv: "1a3a3db574c0d3c3e7b35f1370f467a4"
server_key_version: "encryptionkey@0"
I20260810 07:28:31.161512 6016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:31.184456 6016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:31.188174 6016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:31.188359 6016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:31.196341 6016 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:43331
I20260810 07:28:31.196668 6135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:43331 every 8 connection(s)
I20260810 07:28:31.197592 6016 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:28:31.198252 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 6016
I20260810 07:28:31.198879 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:46095
--local_ip_for_outbound_sockets=127.12.157.193
--tserver_master_addrs=127.12.157.254:43331
--webserver_port=40987
--webserver_interface=127.12.157.193
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:34025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:28:31.203246 6136 sys_catalog.cc:263] Verifying existing consensus state
I20260810 07:28:31.206171 6136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7: Bootstrap starting.
I20260810 07:28:31.233733 6136 log.cc:824] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:31.244568 6136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 07:28:31.245133 6136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7: Bootstrap complete.
I20260810 07:28:31.252717 6136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb300bc5b3c41a1811369fec9238ce7" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 43331 } }
I20260810 07:28:31.253449 6136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bb300bc5b3c41a1811369fec9238ce7, State: Initialized, Role: FOLLOWER
I20260810 07:28:31.254086 6136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb300bc5b3c41a1811369fec9238ce7" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 43331 } }
I20260810 07:28:31.254287 6136 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:28:31.254405 6136 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:28:31.254554 6136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [term 1 FOLLOWER]: Advancing to term 2
I20260810 07:28:31.257102 6136 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb300bc5b3c41a1811369fec9238ce7" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 43331 } }
I20260810 07:28:31.257524 6136 leader_election.cc:302] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bb300bc5b3c41a1811369fec9238ce7; no voters:
I20260810 07:28:31.257897 6136 leader_election.cc:288] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 07:28:31.257995 6144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 07:28:31.258237 6144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [term 2 LEADER]: Becoming Leader. State: Replica: 3bb300bc5b3c41a1811369fec9238ce7, State: Running, Role: LEADER
I20260810 07:28:31.258566 6144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb300bc5b3c41a1811369fec9238ce7" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 43331 } }
I20260810 07:28:31.259366 6136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:28:31.259660 6144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bb300bc5b3c41a1811369fec9238ce7. Latest consensus state: current_term: 2 leader_uuid: "3bb300bc5b3c41a1811369fec9238ce7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb300bc5b3c41a1811369fec9238ce7" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 43331 } } }
I20260810 07:28:31.259804 6144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:31.260193 6145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3bb300bc5b3c41a1811369fec9238ce7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb300bc5b3c41a1811369fec9238ce7" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 43331 } } }
I20260810 07:28:31.260361 6145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:31.265152 6171 catalog_manager.cc:1259] Loaded cluster ID: 66c0b25916ae4849a0ffc05020300c0d
I20260810 07:28:31.265226 6171 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7: loading cluster ID for follower catalog manager: success
I20260810 07:28:31.267401 6171 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7: acquiring CA information for follower catalog manager: success
I20260810 07:28:31.268705 6171 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 07:28:31.268855 6151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:28:31.271018 6151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:28:31.271562 6151 catalog_manager.cc:1259] Loaded cluster ID: 66c0b25916ae4849a0ffc05020300c0d
I20260810 07:28:31.271642 6151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:28:31.272240 6151 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:28:31.272720 6151 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7: Loaded TSK: 0
I20260810 07:28:31.285718 6151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7: Generated new TSK 1
I20260810 07:28:31.286034 6151 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 07:28:31.352510 6140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:31.352839 6140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:31.352891 6140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:31.357528 6140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:31.357620 6140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:31.357733 6140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:28:31.364262 6140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:34025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:46095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=40987
--tserver_master_addrs=127.12.157.254:43331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:28:31.365520 6140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:31.366802 6140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:31.377766 6187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:31.377782 6188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:31.377956 6140 server_base.cc:1034] running on GCE node
W20260810 07:28:31.377767 6190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:31.378391 6140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:31.379122 6140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:31.380306 6140 hybrid_clock.cc:648] HybridClock initialized: now 1786346911380278 us; error 39 us; skew 500 ppm
Aug 10 07:28:31 dist-test-slave-ffrd krb5kdc[5100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346911, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:31.383209 6140 init.cc:377] Logged in from keytab as kudu/127.12.157.193@KRBTEST.COM (short username kudu)
I20260810 07:28:31.384080 6140 webserver.cc:460] Webserver started at http://127.12.157.193:40987/ using document root <none> and password file <none>
I20260810 07:28:31.384636 6140 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:31.384752 6140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:31.389417 6140 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:28:31.391598 6204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:31.392874 6140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 07:28:31.393044 6140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9"
format_stamp: "Formatted at 2026-08-10 07:28:09 on dist-test-slave-ffrd"
server_key: "b4cb964d9b52f43ddb29c25bb7af60df"
server_key_iv: "33514581e8f8f922a92748b9c791f8ee"
server_key_version: "encryptionkey@0"
I20260810 07:28:31.393585 6140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:31.434122 6140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:31.437191 6140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:31.437407 6140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:31.438026 6140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:31.439172 6213 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 07:28:31.443369 6140 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 07:28:31.443454 6140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260810 07:28:31.443537 6140 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 07:28:31.445238 6140 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 07:28:31.445302 6140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260810 07:28:31.445611 6213 tablet_bootstrap.cc:492] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9: Bootstrap starting.
I20260810 07:28:31.455950 6140 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:46095
I20260810 07:28:31.456054 6376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:46095 every 8 connection(s)
I20260810 07:28:31.456982 6140 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:28:31.458684 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 6140
I20260810 07:28:31.459158 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:38525
--local_ip_for_outbound_sockets=127.12.157.194
--tserver_master_addrs=127.12.157.254:43331
--webserver_port=34893
--webserver_interface=127.12.157.194
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:34025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:28:31.468392 6213 log.cc:824] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9: Log is configured to *not* fsync() on all Append() calls
Aug 10 07:28:31 dist-test-slave-ffrd krb5kdc[5100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346911, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:28:31.477303 6213 tablet_bootstrap.cc:492] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 07:28:31.477317 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.460849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:41887 (local address 127.12.157.254:43331)
0810 07:28:31.461488 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:31.461517 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:31.461565 (+ 48us) server_negotiation.cc:407] Connection header received
0810 07:28:31.461878 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:31.461900 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:31.462400 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:31.462829 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:31.464014 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.465221 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.466924 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.467571 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.467750 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.471167 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:31.471175 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:31.471190 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:31.471224 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:31.473497 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:31.474216 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:31.474222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:31.474227 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:31.474390 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:31.474850 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:31.474853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:31.474855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:31.475319 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:31.475453 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:31.476170 (+ 717us) server_negotiation.cc:299] Negotiation successful
0810 07:28:31.476473 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":154,"threads_started":1}
I20260810 07:28:31.477347 6379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.459223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.193:41887)
0810 07:28:31.459876 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:31.459940 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:31.461322 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:31.462868 (+ 1546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:31.462876 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:31.463225 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:28:31.463674 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.463688 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.465408 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.465411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.466777 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.466794 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.467730 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.467737 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.467982 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.468734 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:28:31.468763 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:28:31.470924 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:28:31.473638 (+ 2714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:31.473647 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:31.473665 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:31.474076 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:31.474511 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:31.474517 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:31.474521 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:31.474724 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:31.475460 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:28:31.475470 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:28:31.475814 (+ 344us) client_negotiation.cc:769] Sending connection context
0810 07:28:31.476034 (+ 220us) client_negotiation.cc:240] Negotiation successful
0810 07:28:31.476377 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":187,"threads_started":1}
I20260810 07:28:31.477795 6213 tablet_bootstrap.cc:492] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9: Bootstrap complete.
I20260810 07:28:31.479395 6213 ts_tablet_manager.cc:1397] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9: Time spent bootstrapping tablet: real 0.034s user 0.017s sys 0.014s
I20260810 07:28:31.480123 6377 heartbeater.cc:347] Connected to a master server at 127.12.157.254:43331
I20260810 07:28:31.480355 6377 heartbeater.cc:464] Registering TS with master...
I20260810 07:28:31.480878 6377 heartbeater.cc:511] Master 127.12.157.254:43331 requested a full tablet report, sending...
I20260810 07:28:31.482300 6213 raft_consensus.cc:348] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46095 } } peers { permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 } } peers { permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 } }
I20260810 07:28:31.482848 6213 raft_consensus.cc:729] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294a3cf7eacc474dbec76b0bcd8fa9f9, State: Initialized, Role: FOLLOWER
I20260810 07:28:31.483343 6213 consensus_queue.cc:260] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46095 } } peers { permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 } } peers { permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 } }
I20260810 07:28:31.484275 6073 ts_manager.cc:194] Registered new tserver with Master: 294a3cf7eacc474dbec76b0bcd8fa9f9 (127.12.157.193:46095)
I20260810 07:28:31.484438 6213 ts_tablet_manager.cc:1428] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260810 07:28:31.490732 6073 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.193@KRBTEST.COM'} at 127.12.157.193:41887
I20260810 07:28:31.493732 6377 heartbeater.cc:503] Master 127.12.157.254:43331 was elected leader, sending a full tablet report...
I20260810 07:28:31.502251 6387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.488462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:46095 (local address 127.0.0.1:40732)
0810 07:28:31.492116 (+ 3654us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:31.492166 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:31.492305 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:31.496496 (+ 4191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:31.496502 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:31.496525 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:31.496731 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.496740 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.498239 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.498243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.499777 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.499888 (+ 111us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.501670 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.501674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.501795 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.501846 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 07:28:31.501957 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 07:28:31.502089 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3456,"thread_start_us":111,"threads_started":1}
I20260810 07:28:31.502692 6389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.489300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40732 (local address 127.12.157.193:46095)
0810 07:28:31.495884 (+ 6584us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:31.495897 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:31.495919 (+ 22us) server_negotiation.cc:407] Connection header received
0810 07:28:31.495987 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:31.495992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:31.496176 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:31.496338 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:31.496872 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.498117 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.500336 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.501320 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.501460 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.501589 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:31.502470 (+ 881us) server_negotiation.cc:299] Negotiation successful
0810 07:28:31.502568 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6486,"thread_start_us":111,"threads_started":1}
I20260810 07:28:31.503684 6276 tablet_service.cc:1514] Processing DeleteTablet for tablet 09c82761a3f7466eaed6fdf871f462c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:28:10 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:40732
I20260810 07:28:31.504598 6390 tablet_replica.cc:330] stopping tablet replica
I20260810 07:28:31.504921 6390 raft_consensus.cc:2229] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:28:31.505254 6390 raft_consensus.cc:2258] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:28:31.506237 6390 ts_tablet_manager.cc:1905] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:28:31.510094 6390 ts_tablet_manager.cc:1918] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:28:31.510193 6390 log.cc:1196] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/wal/wals/09c82761a3f7466eaed6fdf871f462c2
I20260810 07:28:31.510473 6390 ts_tablet_manager.cc:1939] T 09c82761a3f7466eaed6fdf871f462c2 P 294a3cf7eacc474dbec76b0bcd8fa9f9: Deleting consensus metadata
I20260810 07:28:31.512233 6050 catalog_manager.cc:4841] TS 294a3cf7eacc474dbec76b0bcd8fa9f9 (127.12.157.193:46095): tablet 09c82761a3f7466eaed6fdf871f462c2 (table test-table [id=a7a5d1c8e3b74c148c71cde5874398ec]) successfully deleted
W20260810 07:28:31.598093 6381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:31.598376 6381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:31.598452 6381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:31.602413 6381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:31.602501 6381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:31.602591 6381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:28:31.610317 6381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:34025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:38525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=34893
--tserver_master_addrs=127.12.157.254:43331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:28:31.611898 6381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:31.613456 6381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:31.623797 6400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:31.623917 6381 server_base.cc:1034] running on GCE node
W20260810 07:28:31.623849 6396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:31.623790 6398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:31.624452 6381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:31.625293 6381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:31.626500 6381 hybrid_clock.cc:648] HybridClock initialized: now 1786346911626483 us; error 34 us; skew 500 ppm
Aug 10 07:28:31 dist-test-slave-ffrd krb5kdc[5100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346911, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:31.629333 6381 init.cc:377] Logged in from keytab as kudu/127.12.157.194@KRBTEST.COM (short username kudu)
I20260810 07:28:31.630187 6381 webserver.cc:460] Webserver started at http://127.12.157.194:34893/ using document root <none> and password file <none>
I20260810 07:28:31.630757 6381 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:31.630865 6381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:31.634692 6381 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260810 07:28:31.636688 6410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:31.637938 6381 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 07:28:31.638069 6381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "20d0364e3e7845678f58e593357e654c"
format_stamp: "Formatted at 2026-08-10 07:28:10 on dist-test-slave-ffrd"
server_key: "0359780f707658a68d0e03835af0aca9"
server_key_iv: "90b48d5108c86a2862bba1d71b3af763"
server_key_version: "encryptionkey@0"
I20260810 07:28:31.638616 6381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:31.669323 6381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:31.672488 6381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:31.672713 6381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:31.673381 6381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:31.674237 6381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:28:31.674306 6381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:31.674376 6381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:28:31.674423 6381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:31.684116 6381 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:38525
I20260810 07:28:31.684154 6606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:38525 every 8 connection(s)
I20260810 07:28:31.685108 6381 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:28:31.686154 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 6381
I20260810 07:28:31.686647 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:41341
--local_ip_for_outbound_sockets=127.12.157.195
--tserver_master_addrs=127.12.157.254:43331
--webserver_port=34209
--webserver_interface=127.12.157.195
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:34025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 07:28:31 dist-test-slave-ffrd krb5kdc[5100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346911, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:28:31.699774 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.687104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:51103 (local address 127.12.157.254:43331)
0810 07:28:31.687248 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:31.687252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:31.688132 (+ 880us) server_negotiation.cc:407] Connection header received
0810 07:28:31.689553 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:31.689557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:31.689625 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:31.689709 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:31.690900 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.691450 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.692582 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.692852 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.692985 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.695430 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:31.695440 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:31.695443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:31.695466 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:31.697286 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:31.698085 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:31.698088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:31.698090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:31.698166 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:31.698580 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:31.698583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:31.698584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:31.698750 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:31.698826 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:31.699538 (+ 712us) server_negotiation.cc:299] Negotiation successful
0810 07:28:31.699651 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 07:28:31.700804 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.687476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.194:51103)
0810 07:28:31.687979 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:31.688020 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:31.689337 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:31.689895 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:31.689901 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:31.690243 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:28:31.690743 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.690758 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.691582 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.691588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.692451 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.692460 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.693032 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.693036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.693197 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.693795 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:28:31.693812 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:28:31.695243 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:28:31.697440 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:31.697448 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:31.697465 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:31.697973 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:31.698267 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:31.698273 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:31.698276 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:31.698479 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:31.698871 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:28:31.698881 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:28:31.699233 (+ 352us) client_negotiation.cc:769] Sending connection context
0810 07:28:31.699482 (+ 249us) client_negotiation.cc:240] Negotiation successful
0810 07:28:31.699842 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":150,"threads_started":1}
I20260810 07:28:31.702075 6607 heartbeater.cc:347] Connected to a master server at 127.12.157.254:43331
I20260810 07:28:31.702325 6607 heartbeater.cc:464] Registering TS with master...
I20260810 07:28:31.702812 6607 heartbeater.cc:511] Master 127.12.157.254:43331 requested a full tablet report, sending...
I20260810 07:28:31.703841 6073 ts_manager.cc:194] Registered new tserver with Master: 20d0364e3e7845678f58e593357e654c (127.12.157.194:38525)
I20260810 07:28:31.704429 6073 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.194@KRBTEST.COM'} at 127.12.157.194:51103
W20260810 07:28:31.803969 6612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:31.804273 6612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:31.804368 6612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:31.809098 6612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:31.809193 6612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:31.809295 6612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:28:31.814867 6612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:34025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:41341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=34209
--tserver_master_addrs=127.12.157.254:43331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:28:31.816011 6612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:31.817119 6612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:31.827052 6623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:31.827108 6630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:31.827070 6625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:31.827358 6612 server_base.cc:1034] running on GCE node
I20260810 07:28:31.827852 6612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:31.828640 6612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:31.829849 6612 hybrid_clock.cc:648] HybridClock initialized: now 1786346911829821 us; error 36 us; skew 500 ppm
Aug 10 07:28:31 dist-test-slave-ffrd krb5kdc[5100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346911, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:31.832774 6612 init.cc:377] Logged in from keytab as kudu/127.12.157.195@KRBTEST.COM (short username kudu)
I20260810 07:28:31.833809 6612 webserver.cc:460] Webserver started at http://127.12.157.195:34209/ using document root <none> and password file <none>
I20260810 07:28:31.834443 6612 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:31.834563 6612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:31.838726 6612 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260810 07:28:31.841264 6638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:31.842762 6612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 07:28:31.842906 6612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "553eefb7d44940aa9c80f507424b214f"
format_stamp: "Formatted at 2026-08-10 07:28:10 on dist-test-slave-ffrd"
server_key: "720466e9669ae0f8459b5001e5b0878b"
server_key_iv: "dc2b88e2319dbdda653d426e6c3a14df"
server_key_version: "encryptionkey@0"
I20260810 07:28:31.843518 6612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:31.862257 6612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:31.865604 6612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:31.865829 6612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:31.866521 6612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:31.867852 6612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:28:31.867944 6612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:31.868022 6612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:28:31.868078 6612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:31.877923 6612 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:41341
I20260810 07:28:31.877982 6814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:41341 every 8 connection(s)
I20260810 07:28:31.878993 6612 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:28:31.885022 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 6612
Aug 10 07:28:31 dist-test-slave-ffrd krb5kdc[5100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346911, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:28:31.895140 6823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.885764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33218 (local address 127.12.157.254:43331)
0810 07:28:31.886070 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:31.886074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:31.886189 (+ 115us) server_negotiation.cc:407] Connection header received
0810 07:28:31.886320 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:31.886322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:31.886366 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:31.886440 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:31.887381 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.888062 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.889308 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.889545 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.889695 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.891517 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:31.891520 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:31.891522 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:31.891545 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:31.893416 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:31.893859 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:31.893861 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:31.893863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:31.893959 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:31.894251 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:31.894254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:31.894256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:31.894454 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:31.894677 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:31.894824 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 07:28:31.894949 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":93,"threads_started":1}
I20260810 07:28:31.895356 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.881094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:54335 (local address 127.12.157.254:43331)
0810 07:28:31.881246 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:31.881250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:31.881980 (+ 730us) server_negotiation.cc:407] Connection header received
0810 07:28:31.883421 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:31.883424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:31.883486 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:31.883591 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:31.884792 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.885490 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.887297 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.887644 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.887753 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.890634 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:31.890639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:31.890642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:31.890683 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:31.892549 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:31.893316 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:31.893319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:31.893320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:31.893419 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:31.893829 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:31.893834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:31.893836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:31.894087 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:31.894197 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:31.894881 (+ 684us) server_negotiation.cc:299] Negotiation successful
0810 07:28:31.895006 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53,"spinlock_wait_cycles":896}
I20260810 07:28:31.896194 6819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.881327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.195:54335)
0810 07:28:31.881806 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:31.881864 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:31.883183 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:31.883774 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:31.883781 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:31.884152 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:28:31.884641 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.884654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.885663 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.885668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.887151 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.887163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.887813 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.887817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.887976 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.888592 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:28:31.888614 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:28:31.890446 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:28:31.892695 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:31.892706 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:31.892726 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:31.893191 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:31.893561 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:31.893564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:31.893565 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:31.893717 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:31.894200 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:28:31.894226 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:28:31.894579 (+ 353us) client_negotiation.cc:769] Sending connection context
0810 07:28:31.894798 (+ 219us) client_negotiation.cc:240] Negotiation successful
0810 07:28:31.895145 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":114,"threads_started":1}
I20260810 07:28:31.897388 6816 heartbeater.cc:347] Connected to a master server at 127.12.157.254:43331
I20260810 07:28:31.897673 6816 heartbeater.cc:464] Registering TS with master...
I20260810 07:28:31.898206 6816 heartbeater.cc:511] Master 127.12.157.254:43331 requested a full tablet report, sending...
I20260810 07:28:31.899248 6073 ts_manager.cc:194] Registered new tserver with Master: 553eefb7d44940aa9c80f507424b214f (127.12.157.195:41341)
I20260810 07:28:31.899854 6073 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.195@KRBTEST.COM'} at 127.12.157.195:54335
I20260810 07:28:31.909369 12919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:28:31.913627 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.910048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33220 (local address 127.12.157.254:43331)
0810 07:28:31.910192 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:31.910196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:31.910414 (+ 218us) server_negotiation.cc:407] Connection header received
0810 07:28:31.910528 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:31.910530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:31.910640 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:31.910737 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:28:31.911012 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.911515 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.912467 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.912683 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.912789 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.913148 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:28:31.913293 (+ 145us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 07:28:31.913523 (+ 230us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33220: token expired
Metrics: {"server-negotiator.queue_time_us":58}
W20260810 07:28:31.913717 6380 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33220: token expired
W20260810 07:28:31.914112 5947 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.157.254:43331)
I20260810 07:28:31.921255 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.914512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33232 (local address 127.12.157.254:43331)
0810 07:28:31.914647 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:31.914650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:31.914686 (+ 36us) server_negotiation.cc:407] Connection header received
0810 07:28:31.914716 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:31.914718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:31.914755 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:31.914816 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:31.915461 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.915945 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.916724 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.916936 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.917096 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.918395 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:31.918398 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:31.918400 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:31.918423 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:31.919698 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:31.920085 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:31.920088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:31.920089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:31.920172 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:31.920460 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:31.920463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:31.920464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:31.920679 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:31.920802 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:31.921042 (+ 240us) server_negotiation.cc:299] Negotiation successful
0810 07:28:31.921141 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 07:28:31.923841 6073 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:28:31.927345 6073 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:28:31.936501 6276 tablet_service.cc:1467] Processing CreateTablet for tablet 02f621edb8964f8bb656bcb516676298 (DEFAULT_TABLE table=test-table [id=cc2d8da911814bd696d73af46d408c40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:28:31.937186 6276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02f621edb8964f8bb656bcb516676298. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:31.940284 6213 tablet_bootstrap.cc:492] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9: Bootstrap starting.
I20260810 07:28:31.940490 6387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.934960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:38525 (local address 127.0.0.1:34302)
0810 07:28:31.935109 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:31.935129 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:31.935206 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:31.936111 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:31.936115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:31.936130 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:31.936249 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.936254 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.937872 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.937875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.938722 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.938732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.940043 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.940047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.940163 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.940189 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 07:28:31.940269 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 07:28:31.940360 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 07:28:31.940670 6829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.935191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34302 (local address 127.12.157.194:38525)
0810 07:28:31.935512 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:31.935522 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:31.935542 (+ 20us) server_negotiation.cc:407] Connection header received
0810 07:28:31.935654 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:31.935660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:31.935938 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:31.936378 (+ 440us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:31.936583 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.937726 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.938896 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.939741 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.939874 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.940008 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:31.940393 (+ 385us) server_negotiation.cc:299] Negotiation successful
0810 07:28:31.940493 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":129,"threads_started":1}
I20260810 07:28:31.941234 6213 tablet_bootstrap.cc:654] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:31.941506 6830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.935603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:41341 (local address 127.0.0.1:36194)
0810 07:28:31.936004 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:31.936021 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:31.936111 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:31.937272 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:31.937275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:31.937295 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:31.937387 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.937393 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.938911 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.938914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.940032 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.940043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.941120 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.941124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.941219 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.941243 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 07:28:31.941308 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 07:28:31.941374 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":109,"threads_started":1}
I20260810 07:28:31.941607 6831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.935842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36194 (local address 127.12.157.195:41341)
0810 07:28:31.936807 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:31.936819 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:31.936843 (+ 24us) server_negotiation.cc:407] Connection header received
0810 07:28:31.936917 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:31.936922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:31.937122 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:31.937268 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:31.937535 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.938775 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.940140 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.940998 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.941139 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.941297 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:31.941352 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 07:28:31.941446 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":857,"thread_start_us":113,"threads_started":1}
I20260810 07:28:31.942890 6213 tablet_bootstrap.cc:492] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9: No bootstrap required, opened a new log
I20260810 07:28:31.942989 6213 ts_tablet_manager.cc:1397] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 07:28:31.943436 6213 raft_consensus.cc:348] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 } } peers { permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 } } peers { permanent_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46095 } }
I20260810 07:28:31.943569 6213 raft_consensus.cc:374] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:31.943609 6213 raft_consensus.cc:729] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294a3cf7eacc474dbec76b0bcd8fa9f9, State: Initialized, Role: FOLLOWER
I20260810 07:28:31.943744 6213 consensus_queue.cc:260] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 } } peers { permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 } } peers { permanent_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46095 } }
I20260810 07:28:31.944008 6213 ts_tablet_manager.cc:1428] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 07:28:31.943902 6493 tablet_service.cc:1467] Processing CreateTablet for tablet 02f621edb8964f8bb656bcb516676298 (DEFAULT_TABLE table=test-table [id=cc2d8da911814bd696d73af46d408c40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:28:31.944311 6707 tablet_service.cc:1467] Processing CreateTablet for tablet 02f621edb8964f8bb656bcb516676298 (DEFAULT_TABLE table=test-table [id=cc2d8da911814bd696d73af46d408c40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:28:31.945055 6493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02f621edb8964f8bb656bcb516676298. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:31.945415 6707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02f621edb8964f8bb656bcb516676298. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:28:31.952476 6835 tablet_bootstrap.cc:492] T 02f621edb8964f8bb656bcb516676298 P 553eefb7d44940aa9c80f507424b214f: Bootstrap starting.
I20260810 07:28:31.952474 6834 tablet_bootstrap.cc:492] T 02f621edb8964f8bb656bcb516676298 P 20d0364e3e7845678f58e593357e654c: Bootstrap starting.
I20260810 07:28:31.954535 6835 tablet_bootstrap.cc:654] T 02f621edb8964f8bb656bcb516676298 P 553eefb7d44940aa9c80f507424b214f: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:31.954974 6834 tablet_bootstrap.cc:654] T 02f621edb8964f8bb656bcb516676298 P 20d0364e3e7845678f58e593357e654c: Neither blocks nor log segments found. Creating new log.
I20260810 07:28:31.955703 6835 log.cc:824] T 02f621edb8964f8bb656bcb516676298 P 553eefb7d44940aa9c80f507424b214f: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:31.955729 6834 log.cc:824] T 02f621edb8964f8bb656bcb516676298 P 20d0364e3e7845678f58e593357e654c: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:31.957864 6835 tablet_bootstrap.cc:492] T 02f621edb8964f8bb656bcb516676298 P 553eefb7d44940aa9c80f507424b214f: No bootstrap required, opened a new log
I20260810 07:28:31.958045 6834 tablet_bootstrap.cc:492] T 02f621edb8964f8bb656bcb516676298 P 20d0364e3e7845678f58e593357e654c: No bootstrap required, opened a new log
I20260810 07:28:31.958069 6835 ts_tablet_manager.cc:1397] T 02f621edb8964f8bb656bcb516676298 P 553eefb7d44940aa9c80f507424b214f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 07:28:31.958246 6834 ts_tablet_manager.cc:1397] T 02f621edb8964f8bb656bcb516676298 P 20d0364e3e7845678f58e593357e654c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
W20260810 07:28:31.958612 6378 tablet.cc:2369] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:28:31.958639 6386 raft_consensus.cc:482] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:28:31.958809 6386 raft_consensus.cc:504] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 } } peers { permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 } } peers { permanent_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46095 } }
I20260810 07:28:31.960021 6386 leader_election.cc:288] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20d0364e3e7845678f58e593357e654c (127.12.157.194:38525), 553eefb7d44940aa9c80f507424b214f (127.12.157.195:41341)
I20260810 07:28:31.962230 6835 raft_consensus.cc:348] T 02f621edb8964f8bb656bcb516676298 P 553eefb7d44940aa9c80f507424b214f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 } } peers { permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 } } peers { permanent_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46095 } }
I20260810 07:28:31.962525 6835 raft_consensus.cc:374] T 02f621edb8964f8bb656bcb516676298 P 553eefb7d44940aa9c80f507424b214f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:31.962577 6835 raft_consensus.cc:729] T 02f621edb8964f8bb656bcb516676298 P 553eefb7d44940aa9c80f507424b214f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 553eefb7d44940aa9c80f507424b214f, State: Initialized, Role: FOLLOWER
I20260810 07:28:31.963063 6835 consensus_queue.cc:260] T 02f621edb8964f8bb656bcb516676298 P 553eefb7d44940aa9c80f507424b214f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 } } peers { permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 } } peers { permanent_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46095 } }
I20260810 07:28:31.964044 6816 heartbeater.cc:503] Master 127.12.157.254:43331 was elected leader, sending a full tablet report...
I20260810 07:28:31.964064 6835 ts_tablet_manager.cc:1428] T 02f621edb8964f8bb656bcb516676298 P 553eefb7d44940aa9c80f507424b214f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 07:28:31.963904 6834 raft_consensus.cc:348] T 02f621edb8964f8bb656bcb516676298 P 20d0364e3e7845678f58e593357e654c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 } } peers { permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 } } peers { permanent_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46095 } }
I20260810 07:28:31.964183 6834 raft_consensus.cc:374] T 02f621edb8964f8bb656bcb516676298 P 20d0364e3e7845678f58e593357e654c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:28:31.964244 6834 raft_consensus.cc:729] T 02f621edb8964f8bb656bcb516676298 P 20d0364e3e7845678f58e593357e654c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20d0364e3e7845678f58e593357e654c, State: Initialized, Role: FOLLOWER
I20260810 07:28:31.964682 6834 consensus_queue.cc:260] T 02f621edb8964f8bb656bcb516676298 P 20d0364e3e7845678f58e593357e654c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 } } peers { permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 } } peers { permanent_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46095 } }
I20260810 07:28:31.965412 6607 heartbeater.cc:503] Master 127.12.157.254:43331 was elected leader, sending a full tablet report...
I20260810 07:28:31.965747 6834 ts_tablet_manager.cc:1428] T 02f621edb8964f8bb656bcb516676298 P 20d0364e3e7845678f58e593357e654c: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260810 07:28:31.966382 6379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.960515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:41341 (local address 127.12.157.193:60875)
0810 07:28:31.960868 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:31.960884 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:31.960998 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:31.961457 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:31.961460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:31.961482 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:31.961611 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.961621 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.962551 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.962555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.963770 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.963794 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.965896 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.965902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.966028 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.966053 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 07:28:31.966134 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 07:28:31.966223 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":100,"threads_started":1}
I20260810 07:28:31.966513 6831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.960882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:60875 (local address 127.12.157.195:41341)
0810 07:28:31.961027 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:31.961034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:31.961051 (+ 17us) server_negotiation.cc:407] Connection header received
0810 07:28:31.961101 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:31.961105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:31.961174 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:31.961250 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:31.961744 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.962445 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.965040 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.965630 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.965750 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.965801 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:31.966295 (+ 494us) server_negotiation.cc:299] Negotiation successful
0810 07:28:31.966375 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 07:28:31.967111 6736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02f621edb8964f8bb656bcb516676298" candidate_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "553eefb7d44940aa9c80f507424b214f" is_pre_election: true
I20260810 07:28:31.967383 6736 raft_consensus.cc:2454] T 02f621edb8964f8bb656bcb516676298 P 553eefb7d44940aa9c80f507424b214f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294a3cf7eacc474dbec76b0bcd8fa9f9 in term 0.
I20260810 07:28:31.967872 6207 leader_election.cc:302] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 294a3cf7eacc474dbec76b0bcd8fa9f9, 553eefb7d44940aa9c80f507424b214f; no voters:
I20260810 07:28:31.968529 6386 raft_consensus.cc:2781] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:28:31.968631 6386 raft_consensus.cc:482] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:28:31.968683 6386 raft_consensus.cc:3037] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:31.969540 6829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.960333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:45873 (local address 127.12.157.194:38525)
0810 07:28:31.960467 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:31.960472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:31.961221 (+ 749us) server_negotiation.cc:407] Connection header received
0810 07:28:31.961316 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:31.961319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:31.961370 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:31.961457 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:28:31.962303 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.963018 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.968138 (+ 5120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.968763 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.968861 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.968960 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:31.969339 (+ 379us) server_negotiation.cc:299] Negotiation successful
0810 07:28:31.969423 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 07:28:31.969878 6840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.960228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:38525 (local address 127.12.157.193:45873)
0810 07:28:31.961127 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:31.961140 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:31.961225 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:31.961652 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:31.961658 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:31.961683 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:28:31.962190 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.962203 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.966644 (+ 4441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.966647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.967977 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:31.967987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.969079 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.969082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:31.969176 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.969198 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 07:28:31.969661 (+ 463us) client_negotiation.cc:240] Negotiation successful
0810 07:28:31.969740 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528}
I20260810 07:28:31.970165 6386 raft_consensus.cc:504] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 } } peers { permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 } } peers { permanent_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46095 } }
I20260810 07:28:31.970556 6386 leader_election.cc:288] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [CANDIDATE]: Term 1 election: Requested vote from peers 20d0364e3e7845678f58e593357e654c (127.12.157.194:38525), 553eefb7d44940aa9c80f507424b214f (127.12.157.195:41341)
I20260810 07:28:31.970749 6530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02f621edb8964f8bb656bcb516676298" candidate_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20d0364e3e7845678f58e593357e654c" is_pre_election: true
I20260810 07:28:31.970937 6736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02f621edb8964f8bb656bcb516676298" candidate_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "553eefb7d44940aa9c80f507424b214f"
I20260810 07:28:31.971028 6736 raft_consensus.cc:3037] T 02f621edb8964f8bb656bcb516676298 P 553eefb7d44940aa9c80f507424b214f [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:31.971046 6529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02f621edb8964f8bb656bcb516676298" candidate_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20d0364e3e7845678f58e593357e654c"
I20260810 07:28:31.971046 6530 raft_consensus.cc:2454] T 02f621edb8964f8bb656bcb516676298 P 20d0364e3e7845678f58e593357e654c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 294a3cf7eacc474dbec76b0bcd8fa9f9 in term 0.
I20260810 07:28:31.972260 6736 raft_consensus.cc:2454] T 02f621edb8964f8bb656bcb516676298 P 553eefb7d44940aa9c80f507424b214f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 294a3cf7eacc474dbec76b0bcd8fa9f9 in term 1.
I20260810 07:28:31.972544 6207 leader_election.cc:302] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 294a3cf7eacc474dbec76b0bcd8fa9f9, 553eefb7d44940aa9c80f507424b214f; no voters: 20d0364e3e7845678f58e593357e654c
I20260810 07:28:31.972711 6386 raft_consensus.cc:2781] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:28:31.972994 6386 raft_consensus.cc:686] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 1 LEADER]: Becoming Leader. State: Replica: 294a3cf7eacc474dbec76b0bcd8fa9f9, State: Running, Role: LEADER
I20260810 07:28:31.973263 6386 consensus_queue.cc:237] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 } } peers { permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 } } peers { permanent_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46095 } }
I20260810 07:28:31.976372 6072 catalog_manager.cc:5495] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 294a3cf7eacc474dbec76b0bcd8fa9f9 (127.12.157.193). New cstate: current_term: 1 leader_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46095 } health_report { overall_health: HEALTHY } } }
I20260810 07:28:31.996387 6389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:31.992872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40740 (local address 127.12.157.193:46095)
0810 07:28:31.993029 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:31.993033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:31.993047 (+ 14us) server_negotiation.cc:407] Connection header received
0810 07:28:31.993206 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:31.993208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:31.993267 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:31.993357 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:28:31.993556 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.994070 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.994917 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:31.995127 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:31.995212 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:31.995477 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:28:31.995938 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:28:31.996027 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:31.996218 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 07:28:31.996275 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 07:28:32.000512 6530 raft_consensus.cc:3037] T 02f621edb8964f8bb656bcb516676298 P 20d0364e3e7845678f58e593357e654c [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:28:32.000659 6736 raft_consensus.cc:1261] T 02f621edb8964f8bb656bcb516676298 P 553eefb7d44940aa9c80f507424b214f [term 1 FOLLOWER]: Refusing update from remote peer 294a3cf7eacc474dbec76b0bcd8fa9f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:28:32.001479 6386 consensus_queue.cc:1035] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:28:32.002255 6530 raft_consensus.cc:1261] T 02f621edb8964f8bb656bcb516676298 P 20d0364e3e7845678f58e593357e654c [term 1 FOLLOWER]: Refusing update from remote peer 294a3cf7eacc474dbec76b0bcd8fa9f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:28:32.002844 6386 consensus_queue.cc:1035] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:28:32.009443 6847 mvcc.cc:205] Tried to move back new op lower bound from 7316876951547994112 to 7316876951445381120. Current Snapshot: MvccSnapshot[applied={T|T < 7316876951547994112}]
I20260810 07:28:32.013229 6848 mvcc.cc:205] Tried to move back new op lower bound from 7316876951547994112 to 7316876951445381120. Current Snapshot: MvccSnapshot[applied={T|T < 7316876951547994112}]
I20260810 07:28:32.013775 6849 mvcc.cc:205] Tried to move back new op lower bound from 7316876951547994112 to 7316876951445381120. Current Snapshot: MvccSnapshot[applied={T|T < 7316876951547994112 or (T in {7316876951547994112})}]
I20260810 07:28:32.017832 6073 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:28:32.018011 6073 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:28:32.020579 6073 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7: Sending DeleteTablet for 3 replicas of tablet 02f621edb8964f8bb656bcb516676298
I20260810 07:28:32.021379 6493 tablet_service.cc:1514] Processing DeleteTablet for tablet 02f621edb8964f8bb656bcb516676298 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:28:32 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:34302
I20260810 07:28:32.021504 6276 tablet_service.cc:1514] Processing DeleteTablet for tablet 02f621edb8964f8bb656bcb516676298 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:28:32 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:40732
I20260810 07:28:32.021932 6707 tablet_service.cc:1514] Processing DeleteTablet for tablet 02f621edb8964f8bb656bcb516676298 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:28:32 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:36194
I20260810 07:28:32.022423 6860 tablet_replica.cc:330] T 02f621edb8964f8bb656bcb516676298 P 553eefb7d44940aa9c80f507424b214f: stopping tablet replica
I20260810 07:28:32.022588 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 6140
I20260810 07:28:32.028887 6859 tablet_replica.cc:330] T 02f621edb8964f8bb656bcb516676298 P 20d0364e3e7845678f58e593357e654c: stopping tablet replica
I20260810 07:28:32.029264 6859 raft_consensus.cc:2229] T 02f621edb8964f8bb656bcb516676298 P 20d0364e3e7845678f58e593357e654c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:28:32.029601 6859 raft_consensus.cc:2258] T 02f621edb8964f8bb656bcb516676298 P 20d0364e3e7845678f58e593357e654c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:28:32.033910 6860 raft_consensus.cc:2229] T 02f621edb8964f8bb656bcb516676298 P 553eefb7d44940aa9c80f507424b214f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:28:32.034284 6860 raft_consensus.cc:2258] T 02f621edb8964f8bb656bcb516676298 P 553eefb7d44940aa9c80f507424b214f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:28:32.035379 6859 ts_tablet_manager.cc:1905] T 02f621edb8964f8bb656bcb516676298 P 20d0364e3e7845678f58e593357e654c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:28:32.035889 6860 ts_tablet_manager.cc:1905] T 02f621edb8964f8bb656bcb516676298 P 553eefb7d44940aa9c80f507424b214f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:28:32.038187 6860 ts_tablet_manager.cc:1918] T 02f621edb8964f8bb656bcb516676298 P 553eefb7d44940aa9c80f507424b214f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:28:32.038309 6860 log.cc:1196] T 02f621edb8964f8bb656bcb516676298 P 553eefb7d44940aa9c80f507424b214f: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/wal/wals/02f621edb8964f8bb656bcb516676298
I20260810 07:28:32.038640 6860 ts_tablet_manager.cc:1939] T 02f621edb8964f8bb656bcb516676298 P 553eefb7d44940aa9c80f507424b214f: Deleting consensus metadata
I20260810 07:28:32.039057 6859 ts_tablet_manager.cc:1918] T 02f621edb8964f8bb656bcb516676298 P 20d0364e3e7845678f58e593357e654c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:28:32.039167 6859 log.cc:1196] T 02f621edb8964f8bb656bcb516676298 P 20d0364e3e7845678f58e593357e654c: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/wal/wals/02f621edb8964f8bb656bcb516676298
I20260810 07:28:32.039512 6859 ts_tablet_manager.cc:1939] T 02f621edb8964f8bb656bcb516676298 P 20d0364e3e7845678f58e593357e654c: Deleting consensus metadata
I20260810 07:28:32.039994 6050 catalog_manager.cc:4841] TS 553eefb7d44940aa9c80f507424b214f (127.12.157.195:41341): tablet 02f621edb8964f8bb656bcb516676298 (table test-table [id=cc2d8da911814bd696d73af46d408c40]) successfully deleted
I20260810 07:28:32.040789 6050 catalog_manager.cc:4841] TS 20d0364e3e7845678f58e593357e654c (127.12.157.194:38525): tablet 02f621edb8964f8bb656bcb516676298 (table test-table [id=cc2d8da911814bd696d73af46d408c40]) successfully deleted
W20260810 07:28:32.042845 6050 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.157.193:46095 (error 108)
I20260810 07:28:32.043064 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 6381
I20260810 07:28:32.053658 6830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:32.053272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:46095 (local address 127.0.0.1:40754)
0810 07:28:32.053428 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:32.053531 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.193:46095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260810 07:28:32.053913 6050 catalog_manager.cc:4584] TS 294a3cf7eacc474dbec76b0bcd8fa9f9 (127.12.157.193:46095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 02f621edb8964f8bb656bcb516676298: Network error: Client connection negotiation failed: client connection to 127.12.157.193:46095: connect: Connection refused (error 111)
I20260810 07:28:32.059605 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 6612
I20260810 07:28:32.076138 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 6016
I20260810 07:28:52.097402 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:43331
--webserver_interface=127.12.157.254
--webserver_port=42839
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:34025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:43331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:28:52.213151 6862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:52.213456 6862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:52.213500 6862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:52.217584 6862 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 07:28:52.217650 6862 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 07:28:52.217666 6862 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 07:28:52.217681 6862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:28:52.217700 6862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:52.217718 6862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:52.217734 6862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:28:52.217749 6862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:28:52.224012 6862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:34025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:43331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:43331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=42839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:28:52.225253 6862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:52.226425 6862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:52.235862 6872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:52.235843 6870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:52.235966 6862 server_base.cc:1034] running on GCE node
W20260810 07:28:52.235843 6876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:52.236742 6862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:52.237926 6862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:52.239095 6862 hybrid_clock.cc:648] HybridClock initialized: now 1786346932239064 us; error 45 us; skew 500 ppm
Aug 10 07:28:52 dist-test-slave-ffrd krb5kdc[5100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346932, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:52.242024 6862 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:28:52.242871 6862 webserver.cc:460] Webserver started at http://127.12.157.254:42839/ using document root <none> and password file <none>
I20260810 07:28:52.243396 6862 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:52.243455 6862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:52.247287 6862 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260810 07:28:52.249526 6889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:52.250793 6862 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260810 07:28:52.250932 6862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "3bb300bc5b3c41a1811369fec9238ce7"
format_stamp: "Formatted at 2026-08-10 07:28:09 on dist-test-slave-ffrd"
server_key: "cd65317d4fb8261f01ed216c32c2f3f5"
server_key_iv: "1a3a3db574c0d3c3e7b35f1370f467a4"
server_key_version: "encryptionkey@0"
I20260810 07:28:52.251555 6862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:52.277485 6862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:52.280776 6862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:52.281050 6862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:52.288722 6862 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:43331
I20260810 07:28:52.289165 6973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:43331 every 8 connection(s)
I20260810 07:28:52.289897 6862 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:28:52.294344 6975 sys_catalog.cc:263] Verifying existing consensus state
I20260810 07:28:52.295789 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 6862
I20260810 07:28:52.296176 6975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7: Bootstrap starting.
I20260810 07:28:52.296283 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:46095
--local_ip_for_outbound_sockets=127.12.157.193
--tserver_master_addrs=127.12.157.254:43331
--webserver_port=40987
--webserver_interface=127.12.157.193
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:34025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:28:52.315659 6975 log.cc:824] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:52.327345 6975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 07:28:52.327761 6975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7: Bootstrap complete.
I20260810 07:28:52.331926 6975 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb300bc5b3c41a1811369fec9238ce7" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 43331 } }
I20260810 07:28:52.332304 6975 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bb300bc5b3c41a1811369fec9238ce7, State: Initialized, Role: FOLLOWER
I20260810 07:28:52.332803 6975 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb300bc5b3c41a1811369fec9238ce7" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 43331 } }
I20260810 07:28:52.332923 6975 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:28:52.333026 6975 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:28:52.333148 6975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [term 2 FOLLOWER]: Advancing to term 3
I20260810 07:28:52.334981 6975 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb300bc5b3c41a1811369fec9238ce7" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 43331 } }
I20260810 07:28:52.335307 6975 leader_election.cc:302] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bb300bc5b3c41a1811369fec9238ce7; no voters:
I20260810 07:28:52.335662 6975 leader_election.cc:288] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 07:28:52.335772 6982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 07:28:52.335981 6982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [term 3 LEADER]: Becoming Leader. State: Replica: 3bb300bc5b3c41a1811369fec9238ce7, State: Running, Role: LEADER
I20260810 07:28:52.336292 6982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb300bc5b3c41a1811369fec9238ce7" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 43331 } }
I20260810 07:28:52.336894 6975 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:28:52.337255 6982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bb300bc5b3c41a1811369fec9238ce7. Latest consensus state: current_term: 3 leader_uuid: "3bb300bc5b3c41a1811369fec9238ce7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb300bc5b3c41a1811369fec9238ce7" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 43331 } } }
I20260810 07:28:52.337391 6982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:52.337767 6983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3bb300bc5b3c41a1811369fec9238ce7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bb300bc5b3c41a1811369fec9238ce7" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 43331 } } }
I20260810 07:28:52.337934 6983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7 [sys.catalog]: This master's current role is: LEADER
I20260810 07:28:52.337839 6991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:28:52.343178 6991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:28:52.343854 6991 catalog_manager.cc:1259] Loaded cluster ID: 66c0b25916ae4849a0ffc05020300c0d
I20260810 07:28:52.343919 6991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:28:52.346482 6991 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:28:52.347213 6991 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7: Loaded TSK: 0
I20260810 07:28:52.347303 6991 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3bb300bc5b3c41a1811369fec9238ce7: Loaded TSK: 1
F20260810 07:28:52.348047 6991 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbea2b9f9ad google::LogMessage::Fail() at ??:0
@ 0x7fbea2ba288a google::LogMessage::SendToLog() at ??:0
@ 0x7fbea2b9f4de google::LogMessage::Flush() at ??:0
@ 0x7fbea2ba0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbea586318d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fbea585bf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fbead5047bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fbead4de99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fbead4df178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fbead520df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fbead53f82b std::function<>::operator()() at ??:0
@ 0x7fbead4deb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fbead4dff3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbead4dc579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fbead520640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fbea4df13d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbea4df1c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fbea4df33f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fbea4de2512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbea64266db start_thread at ??:0
@ 0x7fbea169d61f clone at ??:0
W20260810 07:28:52.435865 6977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:52.436094 6977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:52.436146 6977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:52.440514 6977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:52.440562 6977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:52.440621 6977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:28:52.446290 6977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:34025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:46095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=40987
--tserver_master_addrs=127.12.157.254:43331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:28:52.447443 6977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:52.448511 6977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:52.459086 7052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:52.459086 7048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:52.459355 7054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:52.460807 6977 server_base.cc:1034] running on GCE node
I20260810 07:28:52.461308 6977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:52.462134 6977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:52.463351 6977 hybrid_clock.cc:648] HybridClock initialized: now 1786346932463317 us; error 43 us; skew 500 ppm
Aug 10 07:28:52 dist-test-slave-ffrd krb5kdc[5100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346932, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:52.466866 6977 init.cc:377] Logged in from keytab as kudu/127.12.157.193@KRBTEST.COM (short username kudu)
I20260810 07:28:52.468012 6977 webserver.cc:460] Webserver started at http://127.12.157.193:40987/ using document root <none> and password file <none>
I20260810 07:28:52.468744 6977 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:52.468843 6977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:52.474018 6977 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 07:28:52.476688 7069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:52.478229 6977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:28:52.478371 6977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9"
format_stamp: "Formatted at 2026-08-10 07:28:09 on dist-test-slave-ffrd"
server_key: "b4cb964d9b52f43ddb29c25bb7af60df"
server_key_iv: "33514581e8f8f922a92748b9c791f8ee"
server_key_version: "encryptionkey@0"
I20260810 07:28:52.479085 6977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:52.518203 6977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:52.522336 6977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:52.522544 6977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:52.523285 6977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:52.524833 7084 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 07:28:52.529088 6977 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 07:28:52.529166 6977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260810 07:28:52.529244 6977 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 07:28:52.531455 6977 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 07:28:52.531515 6977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 07:28:52.531867 7084 tablet_bootstrap.cc:492] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9: Bootstrap starting.
I20260810 07:28:52.542935 6977 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:46095
I20260810 07:28:52.543260 7269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:46095 every 8 connection(s)
I20260810 07:28:52.544173 6977 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:28:52.545257 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 6977
I20260810 07:28:52.545787 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:38525
--local_ip_for_outbound_sockets=127.12.157.194
--tserver_master_addrs=127.12.157.254:43331
--webserver_port=34893
--webserver_interface=127.12.157.194
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:34025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:28:52.555290 7084 log.cc:824] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9: Log is configured to *not* fsync() on all Append() calls
I20260810 07:28:52.565299 7084 tablet_bootstrap.cc:492] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 07:28:52.565804 7084 tablet_bootstrap.cc:492] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9: Bootstrap complete.
I20260810 07:28:52.567484 7084 ts_tablet_manager.cc:1397] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9: Time spent bootstrapping tablet: real 0.036s user 0.022s sys 0.010s
I20260810 07:28:52.570916 7084 raft_consensus.cc:348] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 } } peers { permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 } } peers { permanent_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46095 } }
I20260810 07:28:52.571516 7084 raft_consensus.cc:729] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 294a3cf7eacc474dbec76b0bcd8fa9f9, State: Initialized, Role: FOLLOWER
I20260810 07:28:52.572108 7084 consensus_queue.cc:260] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 } } peers { permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 } } peers { permanent_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46095 } }
I20260810 07:28:52.573283 7084 ts_tablet_manager.cc:1428] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
W20260810 07:28:52.626149 7276 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:28:52.549609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.193:58735)
0810 07:28:52.550150 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:52.550205 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:52.551450 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:52.625284 (+ 73834us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":177,"threads_started":1}
W20260810 07:28:52.626778 7270 heartbeater.cc:644] Failed to heartbeat to 127.12.157.254:43331 (0 consecutive failures): Network error: Failed to ping master at 127.12.157.254:43331: Client connection negotiation failed: client connection to 127.12.157.254:43331: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 07:28:52.627655 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:52.627339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.193:33631)
0810 07:28:52.627528 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:52.627595 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260810 07:28:52.628376 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:52.628176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.193:51319)
0810 07:28:52.628286 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:52.628328 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 07:28:52.628568 7270 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 07:28:52.670601 7275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:52.670852 7275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:52.670894 7275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:52.674909 7275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:52.674965 7275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:52.675067 7275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:28:52.680253 7275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:34025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:38525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=34893
--tserver_master_addrs=127.12.157.254:43331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:28:52.681346 7275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:52.682458 7275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:52.692859 7287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:52.693032 7289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:52.692905 7286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:52.693176 7275 server_base.cc:1034] running on GCE node
I20260810 07:28:52.693641 7275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:52.694314 7275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:52.695515 7275 hybrid_clock.cc:648] HybridClock initialized: now 1786346932695500 us; error 28 us; skew 500 ppm
Aug 10 07:28:52 dist-test-slave-ffrd krb5kdc[5100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346932, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:52.698344 7275 init.cc:377] Logged in from keytab as kudu/127.12.157.194@KRBTEST.COM (short username kudu)
I20260810 07:28:52.699241 7275 webserver.cc:460] Webserver started at http://127.12.157.194:34893/ using document root <none> and password file <none>
I20260810 07:28:52.699826 7275 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:52.699918 7275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:52.703742 7275 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260810 07:28:52.705960 7297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:52.707185 7275 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 07:28:52.707326 7275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "20d0364e3e7845678f58e593357e654c"
format_stamp: "Formatted at 2026-08-10 07:28:10 on dist-test-slave-ffrd"
server_key: "0359780f707658a68d0e03835af0aca9"
server_key_iv: "90b48d5108c86a2862bba1d71b3af763"
server_key_version: "encryptionkey@0"
I20260810 07:28:52.707893 7275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:52.720383 7275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:52.723299 7275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:52.723497 7275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:52.724067 7275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:52.724982 7275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:28:52.725054 7275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:52.725118 7275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:28:52.725168 7275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:52.734370 7275 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:38525
I20260810 07:28:52.734404 7477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:38525 every 8 connection(s)
I20260810 07:28:52.735330 7275 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:28:52.738991 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:52.737322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.194:41713)
0810 07:28:52.737776 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:52.737885 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":119,"threads_started":1}
I20260810 07:28:52.739341 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 7275
W20260810 07:28:52.739470 7478 heartbeater.cc:644] Failed to heartbeat to 127.12.157.254:43331 (0 consecutive failures): Network error: Failed to ping master at 127.12.157.254:43331: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
I20260810 07:28:52.739823 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:41341
--local_ip_for_outbound_sockets=127.12.157.195
--tserver_master_addrs=127.12.157.254:43331
--webserver_port=34209
--webserver_interface=127.12.157.195
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:34025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:28:52.740124 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:52.739843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.194:34063)
0810 07:28:52.739982 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:52.740046 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 07:28:52.740803 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:52.740534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.194:60689)
0810 07:28:52.740667 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:52.740726 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 07:28:52.741004 7478 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 07:28:52.857492 7483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:28:52.857756 7483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:28:52.857815 7483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:28:52.861948 7483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:28:52.862020 7483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:28:52.862082 7483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:28:52.867588 7483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:34025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:41341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=34209
--tserver_master_addrs=127.12.157.254:43331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:28:52.868727 7483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:28:52.869840 7483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:28:52.880235 7494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:52.880226 7491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:28:52.880223 7490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:28:52.880270 7483 server_base.cc:1034] running on GCE node
I20260810 07:28:52.881016 7483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:28:52.881734 7483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:28:52.882916 7483 hybrid_clock.cc:648] HybridClock initialized: now 1786346932882901 us; error 29 us; skew 500 ppm
Aug 10 07:28:52 dist-test-slave-ffrd krb5kdc[5100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346932, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:28:52.886344 7483 init.cc:377] Logged in from keytab as kudu/127.12.157.195@KRBTEST.COM (short username kudu)
I20260810 07:28:52.887372 7483 webserver.cc:460] Webserver started at http://127.12.157.195:34209/ using document root <none> and password file <none>
I20260810 07:28:52.887971 7483 fs_manager.cc:359] Metadata directory not provided
I20260810 07:28:52.888038 7483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:28:52.892143 7483 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 07:28:52.894865 7505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:52.896143 7483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 07:28:52.896299 7483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "553eefb7d44940aa9c80f507424b214f"
format_stamp: "Formatted at 2026-08-10 07:28:10 on dist-test-slave-ffrd"
server_key: "720466e9669ae0f8459b5001e5b0878b"
server_key_iv: "dc2b88e2319dbdda653d426e6c3a14df"
server_key_version: "encryptionkey@0"
I20260810 07:28:52.896919 7483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:28:52.929963 7483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:28:52.933387 7483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:28:52.933619 7483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:28:52.934288 7483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:28:52.935256 7483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:28:52.935333 7483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:52.935405 7483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:28:52.935448 7483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:28:52.946250 7483 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:41341
I20260810 07:28:52.946281 7686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:41341 every 8 connection(s)
I20260810 07:28:52.947360 7483 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:28:52.948122 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 7483
I20260810 07:28:52.951828 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:52.949942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.195:47947)
0810 07:28:52.950474 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:52.950583 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":132,"threads_started":1}
W20260810 07:28:52.952286 7690 heartbeater.cc:644] Failed to heartbeat to 127.12.157.254:43331 (0 consecutive failures): Network error: Failed to ping master at 127.12.157.254:43331: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
I20260810 07:28:52.953068 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:52.952782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.195:33713)
0810 07:28:52.952904 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:52.953013 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 07:28:52.953770 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:52.953552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.195:57907)
0810 07:28:52.953665 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:52.953711 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 07:28:52.953956 7690 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 07:28:53.629990 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:53.629360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.193:42209)
0810 07:28:53.629729 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:53.629804 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":96,"threads_started":1}
I20260810 07:28:53.741942 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:53.741546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.194:60831)
0810 07:28:53.741785 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:53.741838 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":69,"threads_started":1}
I20260810 07:28:53.955088 7701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:53.954607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.195:54669)
0810 07:28:53.954878 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:53.954947 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":77,"threads_started":1}
I20260810 07:28:54.175052 7702 raft_consensus.cc:482] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:28:54.175325 7702 raft_consensus.cc:504] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 } } peers { permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 } } peers { permanent_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46095 } }
I20260810 07:28:54.176741 7702 leader_election.cc:288] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20d0364e3e7845678f58e593357e654c (127.12.157.194:38525), 553eefb7d44940aa9c80f507424b214f (127.12.157.195:41341)
Aug 10 07:28:54 dist-test-slave-ffrd krb5kdc[5100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346932, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for kudu/127.12.157.194@KRBTEST.COM
Aug 10 07:28:54 dist-test-slave-ffrd krb5kdc[5100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346932, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for kudu/127.12.157.195@KRBTEST.COM
I20260810 07:28:54.198632 7707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:54.177530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:41341 (local address 127.12.157.193:58361)
0810 07:28:54.177886 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:54.177900 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:54.178000 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:54.179682 (+ 1682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:54.179685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:54.180074 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:28:54.180977 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:54.181001 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:54.183000 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:54.183003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:54.184730 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:54.184742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:54.185818 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:54.185822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:54.186144 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:54.191201 (+ 5057us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:28:54.191234 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:28:54.193819 (+ 2585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:28:54.196175 (+ 2356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:54.196179 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:54.196181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:54.196412 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:54.196779 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:54.196783 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:54.196785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:54.196877 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:54.197585 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:28:54.197592 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:28:54.197847 (+ 255us) client_negotiation.cc:769] Sending connection context
0810 07:28:54.198042 (+ 195us) client_negotiation.cc:240] Negotiation successful
0810 07:28:54.198436 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":446,"spinlock_wait_cycles":35456,"thread_start_us":74,"threads_started":1}
I20260810 07:28:54.198643 7709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:54.177816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:58361 (local address 127.12.157.195:41341)
0810 07:28:54.178251 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:54.178271 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:54.178310 (+ 39us) server_negotiation.cc:407] Connection header received
0810 07:28:54.179105 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:54.179113 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:54.179444 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:54.179707 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:54.181209 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:54.182823 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:54.184872 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:54.185677 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:54.185860 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:54.194083 (+ 8223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:54.194088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:54.194101 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:54.194133 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:54.196035 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:54.196552 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:54.196557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:54.196562 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:54.196676 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:54.197002 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:54.197006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:54.197009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:54.197454 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:54.197600 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:54.198174 (+ 574us) server_negotiation.cc:299] Negotiation successful
0810 07:28:54.198447 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":139,"threads_started":1}
I20260810 07:28:54.198666 7705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:54.177224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:44689 (local address 127.12.157.194:38525)
0810 07:28:54.177588 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0810 07:28:54.177616 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0810 07:28:54.177663 (+ 47us) server_negotiation.cc:407] Connection header received
0810 07:28:54.178630 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:28:54.178642 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:28:54.179091 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:28:54.179435 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:28:54.181209 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:54.182823 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:54.184872 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:28:54.185677 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:28:54.185860 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:54.193870 (+ 8010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:28:54.193877 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:28:54.193892 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:28:54.193926 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:28:54.195830 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:54.196490 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:54.196495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:54.196500 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:54.196619 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:28:54.197005 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:28:54.197009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:28:54.197010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:28:54.197450 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:28:54.197600 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 07:28:54.198175 (+ 575us) server_negotiation.cc:299] Negotiation successful
0810 07:28:54.198469 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":159,"threads_started":1}
I20260810 07:28:54.198701 7704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:54.177100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:38525 (local address 127.12.157.193:44689)
0810 07:28:54.177388 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:54.177413 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 07:28:54.177521 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:28:54.179496 (+ 1975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:28:54.179501 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:28:54.180074 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:28:54.180977 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:54.181001 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:54.183001 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:54.183003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:54.184730 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:28:54.184742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:28:54.185818 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:28:54.185822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:28:54.186145 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:28:54.191201 (+ 5056us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:28:54.191249 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:28:54.193536 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:28:54.195981 (+ 2445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:54.195990 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:54.196016 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:54.196343 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:54.196724 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:28:54.196728 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:28:54.196730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:28:54.196875 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:28:54.197588 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:28:54.197595 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:28:54.197847 (+ 252us) client_negotiation.cc:769] Sending connection context
0810 07:28:54.198042 (+ 195us) client_negotiation.cc:240] Negotiation successful
0810 07:28:54.198534 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":606,"spinlock_wait_cycles":54016,"thread_start_us":100,"threads_started":1}
W20260810 07:28:54.201578 7078 leader_election.cc:341] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 553eefb7d44940aa9c80f507424b214f (127.12.157.195:41341): Not found: Tablet not found: 02f621edb8964f8bb656bcb516676298
W20260810 07:28:54.201784 7078 leader_election.cc:341] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20d0364e3e7845678f58e593357e654c (127.12.157.194:38525): Not found: Tablet not found: 02f621edb8964f8bb656bcb516676298
I20260810 07:28:54.201835 7078 leader_election.cc:302] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294a3cf7eacc474dbec76b0bcd8fa9f9; no voters: 20d0364e3e7845678f58e593357e654c, 553eefb7d44940aa9c80f507424b214f
I20260810 07:28:54.202083 7702 raft_consensus.cc:2726] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 07:28:54.631042 7707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:54.630759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.193:60407)
0810 07:28:54.630889 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:54.630957 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 07:28:54.743139 7711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:54.742733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.194:35921)
0810 07:28:54.742996 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:54.743051 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":81,"threads_started":1}
I20260810 07:28:54.956171 7712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:54.955758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.195:54419)
0810 07:28:54.956018 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:54.956086 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":89,"threads_started":1}
I20260810 07:28:55.632321 7713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:55.631871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.193:33119)
0810 07:28:55.632156 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:55.632220 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":86,"threads_started":1}
I20260810 07:28:55.744417 7715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:55.743980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.194:50895)
0810 07:28:55.744240 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:55.744311 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":79,"threads_started":1}
I20260810 07:28:55.957463 7718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:55.957027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.195:50809)
0810 07:28:55.957283 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:55.957341 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":76,"threads_started":1}
I20260810 07:28:56.212431 7719 raft_consensus.cc:482] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:28:56.212579 7719 raft_consensus.cc:504] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 } } peers { permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 } } peers { permanent_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46095 } }
I20260810 07:28:56.213186 7719 leader_election.cc:288] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20d0364e3e7845678f58e593357e654c (127.12.157.194:38525), 553eefb7d44940aa9c80f507424b214f (127.12.157.195:41341)
W20260810 07:28:56.214159 7078 leader_election.cc:341] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 553eefb7d44940aa9c80f507424b214f (127.12.157.195:41341): Not found: Tablet not found: 02f621edb8964f8bb656bcb516676298
W20260810 07:28:56.214308 7078 leader_election.cc:341] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20d0364e3e7845678f58e593357e654c (127.12.157.194:38525): Not found: Tablet not found: 02f621edb8964f8bb656bcb516676298
I20260810 07:28:56.214372 7078 leader_election.cc:302] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294a3cf7eacc474dbec76b0bcd8fa9f9; no voters: 20d0364e3e7845678f58e593357e654c, 553eefb7d44940aa9c80f507424b214f
I20260810 07:28:56.214563 7719 raft_consensus.cc:2726] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 07:28:56.633908 7720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:56.633448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.193:59087)
0810 07:28:56.633741 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:56.633804 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":100,"threads_started":1}
I20260810 07:28:56.745939 7723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:56.745445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.194:37773)
0810 07:28:56.745755 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:56.745822 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":91,"threads_started":1}
I20260810 07:28:56.958727 7725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:56.958283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.195:40161)
0810 07:28:56.958573 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:56.958630 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":82,"threads_started":1}
I20260810 07:28:57.635802 7727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:57.635202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.193:51411)
0810 07:28:57.635635 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:57.635702 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":106,"threads_started":1}
I20260810 07:28:57.747505 7728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:57.746956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.194:43163)
0810 07:28:57.747290 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:57.747387 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":87,"threads_started":1}
I20260810 07:28:57.960282 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:57.959771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.195:55421)
0810 07:28:57.960092 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:57.960159 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":101,"threads_started":1}
I20260810 07:28:57.987030 7731 raft_consensus.cc:482] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:28:57.987274 7731 raft_consensus.cc:504] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 } } peers { permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 } } peers { permanent_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46095 } }
I20260810 07:28:57.987893 7731 leader_election.cc:288] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20d0364e3e7845678f58e593357e654c (127.12.157.194:38525), 553eefb7d44940aa9c80f507424b214f (127.12.157.195:41341)
W20260810 07:28:57.988888 7078 leader_election.cc:341] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 553eefb7d44940aa9c80f507424b214f (127.12.157.195:41341): Not found: Tablet not found: 02f621edb8964f8bb656bcb516676298
W20260810 07:28:57.989084 7078 leader_election.cc:341] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20d0364e3e7845678f58e593357e654c (127.12.157.194:38525): Not found: Tablet not found: 02f621edb8964f8bb656bcb516676298
I20260810 07:28:57.989154 7078 leader_election.cc:302] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294a3cf7eacc474dbec76b0bcd8fa9f9; no voters: 20d0364e3e7845678f58e593357e654c, 553eefb7d44940aa9c80f507424b214f
I20260810 07:28:57.989328 7731 raft_consensus.cc:2726] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 07:28:58.637161 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:58.636682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.193:33491)
0810 07:28:58.636989 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:58.637053 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":87,"threads_started":1}
I20260810 07:28:58.748802 7735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:58.748362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.194:38621)
0810 07:28:58.748635 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:58.748703 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":78,"threads_started":1}
I20260810 07:28:58.961599 7736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:58.961163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.195:42851)
0810 07:28:58.961443 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:58.961503 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":102,"threads_started":1}
I20260810 07:28:59.638351 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:59.637940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.193:37407)
0810 07:28:59.638191 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:59.638248 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":79,"threads_started":1}
I20260810 07:28:59.750111 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:59.749623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.194:53779)
0810 07:28:59.749956 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:59.750014 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":100,"threads_started":1}
I20260810 07:28:59.963006 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:28:59.962471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.195:34629)
0810 07:28:59.962824 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0810 07:28:59.962889 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":97,"threads_started":1}
I20260810 07:29:00.639633 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:00.639148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.193:52493)
0810 07:29:00.639472 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:00.639533 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":94,"threads_started":1}
I20260810 07:29:00.751508 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:00.751003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.194:59281)
0810 07:29:00.751327 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:00.751390 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":78,"threads_started":1}
I20260810 07:29:00.964658 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:00.963927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.195:38329)
0810 07:29:00.964217 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:00.964276 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":94,"threads_started":1}
I20260810 07:29:01.071969 7754 raft_consensus.cc:482] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:29:01.072165 7754 raft_consensus.cc:504] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 } } peers { permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 } } peers { permanent_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46095 } }
I20260810 07:29:01.072670 7754 leader_election.cc:288] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20d0364e3e7845678f58e593357e654c (127.12.157.194:38525), 553eefb7d44940aa9c80f507424b214f (127.12.157.195:41341)
W20260810 07:29:01.073606 7078 leader_election.cc:341] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 553eefb7d44940aa9c80f507424b214f (127.12.157.195:41341): Not found: Tablet not found: 02f621edb8964f8bb656bcb516676298
W20260810 07:29:01.073769 7078 leader_election.cc:341] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20d0364e3e7845678f58e593357e654c (127.12.157.194:38525): Not found: Tablet not found: 02f621edb8964f8bb656bcb516676298
I20260810 07:29:01.073832 7078 leader_election.cc:302] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294a3cf7eacc474dbec76b0bcd8fa9f9; no voters: 20d0364e3e7845678f58e593357e654c, 553eefb7d44940aa9c80f507424b214f
I20260810 07:29:01.073988 7754 raft_consensus.cc:2726] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 07:29:01.641116 7755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:01.640582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.193:46937)
0810 07:29:01.640869 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:01.640928 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":82,"threads_started":1}
I20260810 07:29:01.752804 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:01.752342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.194:35631)
0810 07:29:01.752607 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:01.752665 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":76,"threads_started":1}
I20260810 07:29:01.967330 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:01.966320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.195:57209)
0810 07:29:01.967072 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:01.967159 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":100,"threads_started":1}
I20260810 07:29:02.642525 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:02.642109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.193:54105)
0810 07:29:02.642368 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:02.642425 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":82,"threads_started":1}
I20260810 07:29:02.754158 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:02.753705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.194:54733)
0810 07:29:02.753995 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:02.754056 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":80,"threads_started":1}
I20260810 07:29:02.968886 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:02.968247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.195:58699)
0810 07:29:02.968685 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:02.968757 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":83,"threads_started":1}
I20260810 07:29:03.421307 7763 raft_consensus.cc:482] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:29:03.421471 7763 raft_consensus.cc:504] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20d0364e3e7845678f58e593357e654c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38525 } } peers { permanent_uuid: "553eefb7d44940aa9c80f507424b214f" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41341 } } peers { permanent_uuid: "294a3cf7eacc474dbec76b0bcd8fa9f9" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 46095 } }
I20260810 07:29:03.421943 7763 leader_election.cc:288] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20d0364e3e7845678f58e593357e654c (127.12.157.194:38525), 553eefb7d44940aa9c80f507424b214f (127.12.157.195:41341)
W20260810 07:29:03.422801 7078 leader_election.cc:341] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20d0364e3e7845678f58e593357e654c (127.12.157.194:38525): Not found: Tablet not found: 02f621edb8964f8bb656bcb516676298
W20260810 07:29:03.422991 7078 leader_election.cc:341] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 553eefb7d44940aa9c80f507424b214f (127.12.157.195:41341): Not found: Tablet not found: 02f621edb8964f8bb656bcb516676298
I20260810 07:29:03.423069 7078 leader_election.cc:302] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 294a3cf7eacc474dbec76b0bcd8fa9f9; no voters: 20d0364e3e7845678f58e593357e654c, 553eefb7d44940aa9c80f507424b214f
I20260810 07:29:03.423249 7763 raft_consensus.cc:2726] T 02f621edb8964f8bb656bcb516676298 P 294a3cf7eacc474dbec76b0bcd8fa9f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 07:29:03.643953 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:03.643462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.193:38967)
0810 07:29:03.643779 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:03.643849 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":83,"threads_started":1}
I20260810 07:29:03.755625 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:03.755178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.194:59163)
0810 07:29:03.755453 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:03.755524 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":78,"threads_started":1}
I20260810 07:29:03.970132 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:03.969706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.195:50157)
0810 07:29:03.969948 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:03.970016 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":78,"threads_started":1}
I20260810 07:29:04.645390 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:04.644878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.193:41277)
0810 07:29:04.645210 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:04.645276 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":135,"threads_started":1}
I20260810 07:29:04.756852 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:04.756414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.194:44793)
0810 07:29:04.756697 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:04.756754 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":93,"threads_started":1}
I20260810 07:29:04.971326 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:04.970901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.195:51845)
0810 07:29:04.971161 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:04.971219 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":83,"threads_started":1}
I20260810 07:29:05.646831 7780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:05.646344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.193:34693)
0810 07:29:05.646656 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:05.646727 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":96,"threads_started":1}
I20260810 07:29:05.758231 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:05.757799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.194:52585)
0810 07:29:05.758070 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:05.758134 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":75,"threads_started":1}
I20260810 07:29:05.972640 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:05.972160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.195:45627)
0810 07:29:05.972459 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:05.972521 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":80,"threads_started":1}
I20260810 07:29:06.648124 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.647635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.193:52097)
0810 07:29:06.647957 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:06.648019 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":82,"threads_started":1}
I20260810 07:29:06.759454 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.759037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.194:60559)
0810 07:29:06.759301 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:06.759356 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":85,"threads_started":1}
I20260810 07:29:06.974022 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:06.973572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.195:42353)
0810 07:29:06.973858 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:06.973921 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":83,"threads_started":1}
I20260810 07:29:07.649729 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:07.649235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.193:43885)
0810 07:29:07.649554 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:07.649617 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":92,"threads_started":1}
I20260810 07:29:07.760751 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:07.760303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.194:41291)
0810 07:29:07.760594 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:07.760654 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":89,"threads_started":1}
I20260810 07:29:07.953222 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 6977
I20260810 07:29:07.971124 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 7275
I20260810 07:29:07.982338 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:07.980201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:43331 (local address 127.12.157.195:40897)
0810 07:29:07.982098 (+ 1897us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:07.982181 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.254:43331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1786,"thread_start_us":791,"threads_started":1}
I20260810 07:29:07.988088 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 7483
2026-08-10T07:29:08Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59867 ms)
[----------] 30 tests from SecurityITest (253018 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:29:08 dist-test-slave-ffrd krb5kdc[7801](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:29:08 dist-test-slave-ffrd krb5kdc[7801](info): set up 2 sockets
Aug 10 07:29:08 dist-test-slave-ffrd krb5kdc[7801](info): commencing operation
krb5kdc: starting...
W20260810 07:29:09.322965 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.297s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:29:09 dist-test-slave-ffrd krb5kdc[7801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:29:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:29:09.462083 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:41231
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:41231
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:09.581533 7821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:09.581795 7821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:09.581835 7821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:09.585901 7821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:09.585958 7821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:09.585978 7821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:09.585995 7821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:09.586046 7821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:09.591246 7821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:41231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:41231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:09.592322 7821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:09.593401 7821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:09.602226 7833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:09.602221 7831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:09.602221 7828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:09.602514 7821 server_base.cc:1034] running on GCE node
I20260810 07:29:09.603096 7821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:09.604233 7821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:09.605444 7821 hybrid_clock.cc:648] HybridClock initialized: now 1786346949605406 us; error 33 us; skew 500 ppm
Aug 10 07:29:09 dist-test-slave-ffrd krb5kdc[7801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346949, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:09.608345 7821 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:29:09.609349 7821 webserver.cc:460] Webserver started at http://127.12.157.254:46879/ using document root <none> and password file <none>
I20260810 07:29:09.609899 7821 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:09.609961 7821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:09.610131 7821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:09.612190 7821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "b253743fe8c54fa2b70a77a8e71f0e34"
format_stamp: "Formatted at 2026-08-10 07:29:09 on dist-test-slave-ffrd"
server_key: "c02df493f99c8cbbe543775f06c4e62f"
server_key_iv: "a1a6cefed038521df9be45fa09ca4602"
server_key_version: "encryptionkey@0"
I20260810 07:29:09.612718 7821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "b253743fe8c54fa2b70a77a8e71f0e34"
format_stamp: "Formatted at 2026-08-10 07:29:09 on dist-test-slave-ffrd"
server_key: "c02df493f99c8cbbe543775f06c4e62f"
server_key_iv: "a1a6cefed038521df9be45fa09ca4602"
server_key_version: "encryptionkey@0"
I20260810 07:29:09.616791 7821 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 07:29:09.619398 7846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:09.620779 7821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 07:29:09.620926 7821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "b253743fe8c54fa2b70a77a8e71f0e34"
format_stamp: "Formatted at 2026-08-10 07:29:09 on dist-test-slave-ffrd"
server_key: "c02df493f99c8cbbe543775f06c4e62f"
server_key_iv: "a1a6cefed038521df9be45fa09ca4602"
server_key_version: "encryptionkey@0"
I20260810 07:29:09.621059 7821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:09.647513 7821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:09.650852 7821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:09.651085 7821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:09.659106 7821 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:41231
I20260810 07:29:09.659121 7938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:41231 every 8 connection(s)
I20260810 07:29:09.660089 7821 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:09.663085 7939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:09.668529 7939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b253743fe8c54fa2b70a77a8e71f0e34: Bootstrap starting.
I20260810 07:29:09.670202 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 7821
I20260810 07:29:09.670328 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:29:09.670626 12919 external_mini_cluster.cc:1428] Setting key ea07deb9d3b6a691cf695d752ceecc05
I20260810 07:29:09.671416 7939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b253743fe8c54fa2b70a77a8e71f0e34: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:09.672298 7939 log.cc:824] T 00000000000000000000000000000000 P b253743fe8c54fa2b70a77a8e71f0e34: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:09.674566 7939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b253743fe8c54fa2b70a77a8e71f0e34: No bootstrap required, opened a new log
I20260810 07:29:09.679001 7939 raft_consensus.cc:348] T 00000000000000000000000000000000 P b253743fe8c54fa2b70a77a8e71f0e34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b253743fe8c54fa2b70a77a8e71f0e34" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41231 } }
I20260810 07:29:09.679217 7939 raft_consensus.cc:374] T 00000000000000000000000000000000 P b253743fe8c54fa2b70a77a8e71f0e34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:09.679250 7939 raft_consensus.cc:729] T 00000000000000000000000000000000 P b253743fe8c54fa2b70a77a8e71f0e34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b253743fe8c54fa2b70a77a8e71f0e34, State: Initialized, Role: FOLLOWER
I20260810 07:29:09.679684 7939 consensus_queue.cc:260] T 00000000000000000000000000000000 P b253743fe8c54fa2b70a77a8e71f0e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b253743fe8c54fa2b70a77a8e71f0e34" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41231 } }
I20260810 07:29:09.679807 7939 raft_consensus.cc:388] T 00000000000000000000000000000000 P b253743fe8c54fa2b70a77a8e71f0e34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:29:09.679889 7939 raft_consensus.cc:482] T 00000000000000000000000000000000 P b253743fe8c54fa2b70a77a8e71f0e34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:29:09.679994 7939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b253743fe8c54fa2b70a77a8e71f0e34 [term 0 FOLLOWER]: Advancing to term 1
Aug 10 07:29:09 dist-test-slave-ffrd krb5kdc[7801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:29:09.681030 7939 raft_consensus.cc:504] T 00000000000000000000000000000000 P b253743fe8c54fa2b70a77a8e71f0e34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b253743fe8c54fa2b70a77a8e71f0e34" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41231 } }
I20260810 07:29:09.681411 7939 leader_election.cc:302] T 00000000000000000000000000000000 P b253743fe8c54fa2b70a77a8e71f0e34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b253743fe8c54fa2b70a77a8e71f0e34; no voters:
I20260810 07:29:09.681788 7939 leader_election.cc:288] T 00000000000000000000000000000000 P b253743fe8c54fa2b70a77a8e71f0e34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:29:09.681857 7948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b253743fe8c54fa2b70a77a8e71f0e34 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:09.682098 7948 raft_consensus.cc:686] T 00000000000000000000000000000000 P b253743fe8c54fa2b70a77a8e71f0e34 [term 1 LEADER]: Becoming Leader. State: Replica: b253743fe8c54fa2b70a77a8e71f0e34, State: Running, Role: LEADER
I20260810 07:29:09.682441 7948 consensus_queue.cc:237] T 00000000000000000000000000000000 P b253743fe8c54fa2b70a77a8e71f0e34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b253743fe8c54fa2b70a77a8e71f0e34" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41231 } }
I20260810 07:29:09.683116 7939 sys_catalog.cc:564] T 00000000000000000000000000000000 P b253743fe8c54fa2b70a77a8e71f0e34 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:09.684170 7951 sys_catalog.cc:455] T 00000000000000000000000000000000 P b253743fe8c54fa2b70a77a8e71f0e34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b253743fe8c54fa2b70a77a8e71f0e34. Latest consensus state: current_term: 1 leader_uuid: "b253743fe8c54fa2b70a77a8e71f0e34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b253743fe8c54fa2b70a77a8e71f0e34" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41231 } } }
I20260810 07:29:09.684296 7951 sys_catalog.cc:458] T 00000000000000000000000000000000 P b253743fe8c54fa2b70a77a8e71f0e34 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:09.684460 7950 sys_catalog.cc:455] T 00000000000000000000000000000000 P b253743fe8c54fa2b70a77a8e71f0e34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b253743fe8c54fa2b70a77a8e71f0e34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b253743fe8c54fa2b70a77a8e71f0e34" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41231 } } }
I20260810 07:29:09.684652 7950 sys_catalog.cc:458] T 00000000000000000000000000000000 P b253743fe8c54fa2b70a77a8e71f0e34 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:09.685249 7962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:09.686861 7946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.671801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59088 (local address 127.12.157.254:41231)
0810 07:29:09.672244 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:09.672261 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:09.672296 (+ 35us) server_negotiation.cc:407] Connection header received
0810 07:29:09.673212 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:09.673232 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:09.673558 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:09.673828 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:09.674796 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.676299 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.677210 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.678010 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.678180 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:09.681049 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:09.681057 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:09.681074 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:09.681107 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:09.683093 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.683564 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.683572 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.683577 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.683725 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.684107 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.684112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.684115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.684686 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:09.684902 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:09.685599 (+ 697us) server_negotiation.cc:299] Negotiation successful
0810 07:29:09.685906 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":152,"threads_started":1}
I20260810 07:29:09.688632 7962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:09.693917 7962 catalog_manager.cc:1347] Generated new cluster ID: 529804e82d3e47abbcfdfb370fbc54e3
I20260810 07:29:09.693986 7962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:09.710637 7962 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:09.711647 7962 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:09.715958 7962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b253743fe8c54fa2b70a77a8e71f0e34: Generated new TSK 0
I20260810 07:29:09.716733 7962 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
I20260810 07:29:09.755898 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:0
--local_ip_for_outbound_sockets=127.12.157.193
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:41231
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:09.874704 7983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:09.874996 7983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:09.875072 7983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:09.879072 7983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:09.879160 7983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:09.879251 7983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:29:09.884675 7983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:41231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:09.885846 7983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:09.886888 7983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:09.896669 7993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:09.896677 8000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:09.896669 7997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:09.897029 7983 server_base.cc:1034] running on GCE node
I20260810 07:29:09.897434 7983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:09.898083 7983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:09.899272 7983 hybrid_clock.cc:648] HybridClock initialized: now 1786346949899240 us; error 46 us; skew 500 ppm
Aug 10 07:29:09 dist-test-slave-ffrd krb5kdc[7801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346949, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:09.902144 7983 init.cc:377] Logged in from keytab as kudu/127.12.157.193@KRBTEST.COM (short username kudu)
I20260810 07:29:09.903189 7983 webserver.cc:460] Webserver started at http://127.12.157.193:39409/ using document root <none> and password file <none>
I20260810 07:29:09.903759 7983 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:09.903853 7983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:09.904116 7983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:09.906168 7983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-0/data/instance:
uuid: "7904b95611f549bca01e22b57d8e6bbd"
format_stamp: "Formatted at 2026-08-10 07:29:09 on dist-test-slave-ffrd"
server_key: "1f0c3f2baef79b7b0eea4143b84d4577"
server_key_iv: "14579456b90075d8f548eac5ad8433d5"
server_key_version: "encryptionkey@0"
I20260810 07:29:09.906751 7983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance:
uuid: "7904b95611f549bca01e22b57d8e6bbd"
format_stamp: "Formatted at 2026-08-10 07:29:09 on dist-test-slave-ffrd"
server_key: "1f0c3f2baef79b7b0eea4143b84d4577"
server_key_iv: "14579456b90075d8f548eac5ad8433d5"
server_key_version: "encryptionkey@0"
I20260810 07:29:09.910490 7983 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 07:29:09.912819 8010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:09.914125 7983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:29:09.914289 7983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "7904b95611f549bca01e22b57d8e6bbd"
format_stamp: "Formatted at 2026-08-10 07:29:09 on dist-test-slave-ffrd"
server_key: "1f0c3f2baef79b7b0eea4143b84d4577"
server_key_iv: "14579456b90075d8f548eac5ad8433d5"
server_key_version: "encryptionkey@0"
I20260810 07:29:09.914405 7983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:09.936609 7983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:09.939860 7983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:09.940083 7983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:09.940701 7983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:29:09.941608 7983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:29:09.941656 7983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:09.941726 7983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:29:09.941771 7983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:09.951449 7983 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:45581
I20260810 07:29:09.951470 8211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:45581 every 8 connection(s)
I20260810 07:29:09.952548 7983 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:29:09.954970 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 7983
I20260810 07:29:09.955111 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance
I20260810 07:29:09.955441 12919 external_mini_cluster.cc:1428] Setting key 3526150184ddb15124c06b6992676f5d
Aug 10 07:29:09 dist-test-slave-ffrd krb5kdc[7801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346949, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.157.194@KRBTEST.COM; defaulting to no policy
I20260810 07:29:09.967774 7946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.954668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:39903 (local address 127.12.157.254:41231)
0810 07:29:09.954814 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:09.954818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:09.955831 (+ 1013us) server_negotiation.cc:407] Connection header received
0810 07:29:09.957503 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:09.957507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:09.957562 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:09.957643 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:09.958797 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.959335 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.960490 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.960894 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.961037 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:09.963743 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:09.963751 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:09.963756 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:09.963791 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:09.965586 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.966154 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.966160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.966165 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.966265 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:09.966623 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:09.966626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:09.966628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:09.966870 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:09.966987 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:09.967530 (+ 543us) server_negotiation.cc:299] Negotiation successful
0810 07:29:09.967645 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 07:29:09.968600 8214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:09.955157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:41231 (local address 127.12.157.193:39903)
0810 07:29:09.955680 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:09.955721 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:09.957288 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:09.957811 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:09.957818 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:09.958169 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:09.958648 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:09.958662 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.959477 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.959481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:09.960358 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:09.960368 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:09.961096 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:09.961099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:09.961259 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:09.961909 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:09.961926 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:09.963557 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:09.965728 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:09.965733 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:09.965745 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:09.966042 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:09.966383 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:09.966385 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:09.966387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:09.966520 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:09.966993 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:09.967002 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:09.967253 (+ 251us) client_negotiation.cc:769] Sending connection context
0810 07:29:09.967422 (+ 169us) client_negotiation.cc:240] Negotiation successful
0810 07:29:09.967678 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":148,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.194@KRBTEST.COM" created.
I20260810 07:29:09.970320 8212 heartbeater.cc:347] Connected to a master server at 127.12.157.254:41231
I20260810 07:29:09.970599 8212 heartbeater.cc:464] Registering TS with master...
I20260810 07:29:09.971134 8212 heartbeater.cc:511] Master 127.12.157.254:41231 requested a full tablet report, sending...
I20260810 07:29:09.972781 7879 ts_manager.cc:194] Registered new tserver with Master: 7904b95611f549bca01e22b57d8e6bbd (127.12.157.193:45581)
I20260810 07:29:09.974491 7879 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.193@KRBTEST.COM'} at 127.12.157.193:39903
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
I20260810 07:29:09.987399 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:0
--local_ip_for_outbound_sockets=127.12.157.194
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:41231
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:10.105631 8217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:10.105933 8217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:10.105983 8217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:10.110162 8217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:10.110234 8217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:10.110297 8217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:29:10.115779 8217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:41231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:10.116889 8217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:10.118130 8217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:10.129412 8225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:10.129412 8229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:10.129494 8224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:10.130259 8217 server_base.cc:1034] running on GCE node
I20260810 07:29:10.131012 8217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:10.131790 8217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:10.133082 8217 hybrid_clock.cc:648] HybridClock initialized: now 1786346950133064 us; error 32 us; skew 500 ppm
Aug 10 07:29:10 dist-test-slave-ffrd krb5kdc[7801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346950, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:10.136533 8217 init.cc:377] Logged in from keytab as kudu/127.12.157.194@KRBTEST.COM (short username kudu)
I20260810 07:29:10.137662 8217 webserver.cc:460] Webserver started at http://127.12.157.194:33261/ using document root <none> and password file <none>
I20260810 07:29:10.138289 8217 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:10.138391 8217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:10.138643 8217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:10.141085 8217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-1/data/instance:
uuid: "f0883e535e1a4903aec5e84fe7ecdab2"
format_stamp: "Formatted at 2026-08-10 07:29:10 on dist-test-slave-ffrd"
server_key: "dccaccdd711a4e5732c05c60568c40e6"
server_key_iv: "a2698f8ff12fd78cd344faccc689c397"
server_key_version: "encryptionkey@0"
I20260810 07:29:10.141731 8217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance:
uuid: "f0883e535e1a4903aec5e84fe7ecdab2"
format_stamp: "Formatted at 2026-08-10 07:29:10 on dist-test-slave-ffrd"
server_key: "dccaccdd711a4e5732c05c60568c40e6"
server_key_iv: "a2698f8ff12fd78cd344faccc689c397"
server_key_version: "encryptionkey@0"
I20260810 07:29:10.145934 8217 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 07:29:10.148869 8239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:10.150633 8217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 07:29:10.150828 8217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "f0883e535e1a4903aec5e84fe7ecdab2"
format_stamp: "Formatted at 2026-08-10 07:29:10 on dist-test-slave-ffrd"
server_key: "dccaccdd711a4e5732c05c60568c40e6"
server_key_iv: "a2698f8ff12fd78cd344faccc689c397"
server_key_version: "encryptionkey@0"
I20260810 07:29:10.151000 8217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:10.175244 8217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:10.178879 8217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:10.179109 8217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:10.179746 8217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:29:10.180634 8217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:29:10.180704 8217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:10.180780 8217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:29:10.180832 8217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:10.192373 8217 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:34019
I20260810 07:29:10.192518 8438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:34019 every 8 connection(s)
I20260810 07:29:10.193411 8217 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:29:10.195727 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 8217
I20260810 07:29:10.195852 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance
I20260810 07:29:10.196144 12919 external_mini_cluster.cc:1428] Setting key f6e0e6f75b30647d18ea764a7ca66acc
Aug 10 07:29:10 dist-test-slave-ffrd krb5kdc[7801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346950, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.157.195@KRBTEST.COM; defaulting to no policy
I20260810 07:29:10.209637 7946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:10.195312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:52511 (local address 127.12.157.254:41231)
0810 07:29:10.195592 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:10.195597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:10.196408 (+ 811us) server_negotiation.cc:407] Connection header received
0810 07:29:10.198004 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:10.198008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:10.198063 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:10.198158 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:10.199600 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.200388 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.202105 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.202533 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.202661 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:10.205503 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:10.205509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:10.205514 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:10.205547 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:10.207325 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:10.207901 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:10.207906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:10.207908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:10.207997 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:10.208426 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:10.208432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:10.208434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:10.208656 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:10.208738 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:10.209397 (+ 659us) server_negotiation.cc:299] Negotiation successful
0810 07:29:10.209511 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260810 07:29:10.210645 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:10.195713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:41231 (local address 127.12.157.194:52511)
0810 07:29:10.196235 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:10.196288 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:10.197797 (+ 1509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:10.198411 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:10.198418 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:10.198913 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:10.199441 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:10.199456 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.200542 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.200548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:10.201965 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:10.201977 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.202688 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.202691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:10.202830 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:10.203499 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:10.203516 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:10.205311 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:10.207466 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:10.207472 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:10.207485 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:10.207755 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:10.208091 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:10.208093 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:10.208094 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:10.208286 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:10.208754 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:10.208759 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:10.209085 (+ 326us) client_negotiation.cc:769] Sending connection context
0810 07:29:10.209285 (+ 200us) client_negotiation.cc:240] Negotiation successful
0810 07:29:10.209668 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":145,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.195@KRBTEST.COM" created.
I20260810 07:29:10.211982 8441 heartbeater.cc:347] Connected to a master server at 127.12.157.254:41231
I20260810 07:29:10.212248 8441 heartbeater.cc:464] Registering TS with master...
I20260810 07:29:10.212774 8441 heartbeater.cc:511] Master 127.12.157.254:41231 requested a full tablet report, sending...
I20260810 07:29:10.214071 7879 ts_manager.cc:194] Registered new tserver with Master: f0883e535e1a4903aec5e84fe7ecdab2 (127.12.157.194:34019)
I20260810 07:29:10.214677 7879 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.194@KRBTEST.COM'} at 127.12.157.194:52511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
I20260810 07:29:10.228797 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:0
--local_ip_for_outbound_sockets=127.12.157.195
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:41231
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:36711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:10.342126 8449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:10.342370 8449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:10.342411 8449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:10.346318 8449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:10.346369 8449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:10.346426 8449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:29:10.351601 8449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:36711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:41231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:10.352546 8449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:10.353611 8449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:10.363745 8465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:10.363714 8456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:10.363718 8462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:10.364151 8449 server_base.cc:1034] running on GCE node
I20260810 07:29:10.364578 8449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:10.365250 8449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:10.366469 8449 hybrid_clock.cc:648] HybridClock initialized: now 1786346950366439 us; error 41 us; skew 500 ppm
Aug 10 07:29:10 dist-test-slave-ffrd krb5kdc[7801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346950, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:10.369685 8449 init.cc:377] Logged in from keytab as kudu/127.12.157.195@KRBTEST.COM (short username kudu)
I20260810 07:29:10.370638 8449 webserver.cc:460] Webserver started at http://127.12.157.195:35949/ using document root <none> and password file <none>
I20260810 07:29:10.371223 8449 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:10.371325 8449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:10.371580 8449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:10.373735 8449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-2/data/instance:
uuid: "f8e6b81932464a7fb592cbbe312ff6a8"
format_stamp: "Formatted at 2026-08-10 07:29:10 on dist-test-slave-ffrd"
server_key: "9d314f9737605717a6a96961b9446e96"
server_key_iv: "c55b995b2133c827b0d3db85fd839c8a"
server_key_version: "encryptionkey@0"
I20260810 07:29:10.374321 8449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance:
uuid: "f8e6b81932464a7fb592cbbe312ff6a8"
format_stamp: "Formatted at 2026-08-10 07:29:10 on dist-test-slave-ffrd"
server_key: "9d314f9737605717a6a96961b9446e96"
server_key_iv: "c55b995b2133c827b0d3db85fd839c8a"
server_key_version: "encryptionkey@0"
I20260810 07:29:10.378422 8449 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 07:29:10.381379 8481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:10.382736 8449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 07:29:10.382876 8449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "f8e6b81932464a7fb592cbbe312ff6a8"
format_stamp: "Formatted at 2026-08-10 07:29:10 on dist-test-slave-ffrd"
server_key: "9d314f9737605717a6a96961b9446e96"
server_key_iv: "c55b995b2133c827b0d3db85fd839c8a"
server_key_version: "encryptionkey@0"
I20260810 07:29:10.383001 8449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:10.394629 8449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:10.398188 8449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:10.398398 8449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:10.399050 8449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:29:10.399936 8449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:29:10.399984 8449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:10.400025 8449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:29:10.400041 8449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:10.409789 8449 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:38431
I20260810 07:29:10.409809 8688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:38431 every 8 connection(s)
I20260810 07:29:10.410750 8449 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:29:10.415998 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 8449
I20260810 07:29:10.416117 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance
I20260810 07:29:10.416466 12919 external_mini_cluster.cc:1428] Setting key b71b65bd1d4a7d3d8c83434b936e44bc
Aug 10 07:29:10 dist-test-slave-ffrd krb5kdc[7801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346950, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:29:10.426431 7946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:10.412835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:40535 (local address 127.12.157.254:41231)
0810 07:29:10.413053 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:10.413057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:10.413939 (+ 882us) server_negotiation.cc:407] Connection header received
0810 07:29:10.415441 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:10.415445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:10.415509 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:10.415623 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:10.417172 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.417830 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.419366 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.419631 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.419751 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:10.422625 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:10.422629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:10.422631 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:10.422657 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:10.424155 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:10.424740 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:10.424744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:10.424746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:10.424848 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:10.425244 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:10.425247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:10.425249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:10.425453 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:10.425566 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:10.426120 (+ 554us) server_negotiation.cc:299] Negotiation successful
0810 07:29:10.426248 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 07:29:10.427292 8691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:10.413258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:41231 (local address 127.12.157.195:40535)
0810 07:29:10.413765 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:10.413811 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:10.415192 (+ 1381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:10.415832 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:10.415839 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:10.416325 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:10.417011 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:10.417043 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.417969 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.417973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:10.419214 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:10.419225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.419798 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.419802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:10.419991 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:10.420684 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:10.420701 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:10.422318 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:10.424285 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:10.424292 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:10.424310 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:10.424602 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:10.424978 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:10.424981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:10.424983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:10.425136 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:10.425566 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:10.425574 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:10.425853 (+ 279us) client_negotiation.cc:769] Sending connection context
0810 07:29:10.426066 (+ 213us) client_negotiation.cc:240] Negotiation successful
0810 07:29:10.426426 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":115,"threads_started":1}
I20260810 07:29:10.428550 8689 heartbeater.cc:347] Connected to a master server at 127.12.157.254:41231
I20260810 07:29:10.428778 8689 heartbeater.cc:464] Registering TS with master...
I20260810 07:29:10.429399 8689 heartbeater.cc:511] Master 127.12.157.254:41231 requested a full tablet report, sending...
I20260810 07:29:10.430648 7879 ts_manager.cc:194] Registered new tserver with Master: f8e6b81932464a7fb592cbbe312ff6a8 (127.12.157.195:38431)
I20260810 07:29:10.431244 7879 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.195@KRBTEST.COM'} at 127.12.157.195:40535
I20260810 07:29:10.442533 12919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:29:10.453001 7946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:10.444690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59094 (local address 127.12.157.254:41231)
0810 07:29:10.444876 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:10.444881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:10.445120 (+ 239us) server_negotiation.cc:407] Connection header received
0810 07:29:10.445283 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:10.445287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:10.445336 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:10.445448 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:10.446189 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.446727 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.447604 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.447837 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.447948 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:10.449517 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:10.449521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:10.449524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:10.449558 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:10.451012 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:10.451620 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:10.451624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:10.451626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:10.451717 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:10.452016 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:10.452020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:10.452022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:10.452268 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:10.452431 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:10.452672 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 07:29:10.452810 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 07:29:10.456630 7879 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:29:10.460225 7879 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:29:10.476996 8709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:10.470822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:45581 (local address 127.0.0.1:53030)
0810 07:29:10.471258 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:10.471317 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:10.471578 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:10.472367 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:10.472374 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:10.472393 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:10.472552 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:10.472559 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.474048 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.474052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:10.475279 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:10.475291 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.476401 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.476405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:10.476515 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:10.476558 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 07:29:10.476675 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 07:29:10.476792 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":106,"threads_started":1}
I20260810 07:29:10.477033 8712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:10.471035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53030 (local address 127.12.157.193:45581)
0810 07:29:10.471459 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:10.471475 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:10.471510 (+ 35us) server_negotiation.cc:407] Connection header received
0810 07:29:10.471777 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:10.471785 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:10.472012 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:10.472171 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:10.472697 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.473919 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.475420 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.476251 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.476393 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:10.476535 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:10.476727 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 07:29:10.476849 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":116,"threads_started":1}
I20260810 07:29:10.477556 8714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:10.471531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:34019 (local address 127.0.0.1:53920)
0810 07:29:10.472283 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:10.472300 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:10.472382 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:10.473121 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:10.473124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:10.473141 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:10.473216 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:10.473221 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.474627 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.474631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:10.476080 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:10.476093 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.477255 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.477260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:10.477346 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:10.477364 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 07:29:10.477406 (+ 42us) client_negotiation.cc:240] Negotiation successful
0810 07:29:10.477454 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":147,"thread_start_us":127,"threads_started":1}
I20260810 07:29:10.478098 8716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:10.472043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53920 (local address 127.12.157.194:34019)
0810 07:29:10.472548 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:10.472567 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:10.472598 (+ 31us) server_negotiation.cc:407] Connection header received
0810 07:29:10.472682 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:10.472687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:10.472923 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:10.473382 (+ 459us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:10.473535 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.474458 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.476224 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.477107 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.477654 (+ 547us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:10.477824 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:10.477874 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 07:29:10.477964 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":110,"threads_started":1}
I20260810 07:29:10.478466 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:10.471984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46332 (local address 127.12.157.195:38431)
0810 07:29:10.472969 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:10.472987 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:10.473011 (+ 24us) server_negotiation.cc:407] Connection header received
0810 07:29:10.473085 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:10.473090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:10.473255 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:10.473407 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:10.473588 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.474464 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.476246 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.477154 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.477275 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:10.477606 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:10.478194 (+ 588us) server_negotiation.cc:299] Negotiation successful
0810 07:29:10.478307 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":852,"thread_start_us":166,"threads_started":1}
I20260810 07:29:10.478080 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:10.471564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:38431 (local address 127.0.0.1:46332)
0810 07:29:10.472567 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:10.472581 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:10.472656 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:10.473396 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:10.473398 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:10.473414 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:10.473490 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:10.473495 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.474611 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.474617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:10.476087 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:10.476093 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.477689 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.477694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:10.477786 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:10.477819 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 07:29:10.477894 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 07:29:10.477967 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"mutex_wait_us":130,"thread_start_us":115,"threads_started":1}
I20260810 07:29:10.481055 8101 tablet_service.cc:1467] Processing CreateTablet for tablet 7275f9f9e7be4e089f4ded93647dec58 (DEFAULT_TABLE table=test-table [id=235e573b1d0e430caf79215891a44f08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:29:10.481280 8324 tablet_service.cc:1467] Processing CreateTablet for tablet 7275f9f9e7be4e089f4ded93647dec58 (DEFAULT_TABLE table=test-table [id=235e573b1d0e430caf79215891a44f08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:29:10.482123 8101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7275f9f9e7be4e089f4ded93647dec58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:10.482317 8324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7275f9f9e7be4e089f4ded93647dec58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:10.482136 8581 tablet_service.cc:1467] Processing CreateTablet for tablet 7275f9f9e7be4e089f4ded93647dec58 (DEFAULT_TABLE table=test-table [id=235e573b1d0e430caf79215891a44f08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:29:10.483119 8581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7275f9f9e7be4e089f4ded93647dec58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:10.489940 8718 tablet_bootstrap.cc:492] T 7275f9f9e7be4e089f4ded93647dec58 P 7904b95611f549bca01e22b57d8e6bbd: Bootstrap starting.
I20260810 07:29:10.490916 8719 tablet_bootstrap.cc:492] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2: Bootstrap starting.
I20260810 07:29:10.492093 8718 tablet_bootstrap.cc:654] T 7275f9f9e7be4e089f4ded93647dec58 P 7904b95611f549bca01e22b57d8e6bbd: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:10.492645 8719 tablet_bootstrap.cc:654] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:10.492930 8718 log.cc:824] T 7275f9f9e7be4e089f4ded93647dec58 P 7904b95611f549bca01e22b57d8e6bbd: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:10.493374 8720 tablet_bootstrap.cc:492] T 7275f9f9e7be4e089f4ded93647dec58 P f8e6b81932464a7fb592cbbe312ff6a8: Bootstrap starting.
I20260810 07:29:10.494053 8719 log.cc:824] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:10.495122 8718 tablet_bootstrap.cc:492] T 7275f9f9e7be4e089f4ded93647dec58 P 7904b95611f549bca01e22b57d8e6bbd: No bootstrap required, opened a new log
I20260810 07:29:10.495325 8718 ts_tablet_manager.cc:1397] T 7275f9f9e7be4e089f4ded93647dec58 P 7904b95611f549bca01e22b57d8e6bbd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 07:29:10.495651 8720 tablet_bootstrap.cc:654] T 7275f9f9e7be4e089f4ded93647dec58 P f8e6b81932464a7fb592cbbe312ff6a8: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:10.496043 8719 tablet_bootstrap.cc:492] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2: No bootstrap required, opened a new log
I20260810 07:29:10.496255 8719 ts_tablet_manager.cc:1397] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260810 07:29:10.496963 8720 log.cc:824] T 7275f9f9e7be4e089f4ded93647dec58 P f8e6b81932464a7fb592cbbe312ff6a8: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:10.498963 8720 tablet_bootstrap.cc:492] T 7275f9f9e7be4e089f4ded93647dec58 P f8e6b81932464a7fb592cbbe312ff6a8: No bootstrap required, opened a new log
I20260810 07:29:10.499166 8720 ts_tablet_manager.cc:1397] T 7275f9f9e7be4e089f4ded93647dec58 P f8e6b81932464a7fb592cbbe312ff6a8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 07:29:10.499164 8718 raft_consensus.cc:348] T 7275f9f9e7be4e089f4ded93647dec58 P 7904b95611f549bca01e22b57d8e6bbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7904b95611f549bca01e22b57d8e6bbd" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 45581 } } peers { permanent_uuid: "f8e6b81932464a7fb592cbbe312ff6a8" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 38431 } } peers { permanent_uuid: "f0883e535e1a4903aec5e84fe7ecdab2" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 34019 } }
I20260810 07:29:10.499475 8718 raft_consensus.cc:374] T 7275f9f9e7be4e089f4ded93647dec58 P 7904b95611f549bca01e22b57d8e6bbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:10.499531 8718 raft_consensus.cc:729] T 7275f9f9e7be4e089f4ded93647dec58 P 7904b95611f549bca01e22b57d8e6bbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7904b95611f549bca01e22b57d8e6bbd, State: Initialized, Role: FOLLOWER
I20260810 07:29:10.499984 8718 consensus_queue.cc:260] T 7275f9f9e7be4e089f4ded93647dec58 P 7904b95611f549bca01e22b57d8e6bbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7904b95611f549bca01e22b57d8e6bbd" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 45581 } } peers { permanent_uuid: "f8e6b81932464a7fb592cbbe312ff6a8" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 38431 } } peers { permanent_uuid: "f0883e535e1a4903aec5e84fe7ecdab2" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 34019 } }
I20260810 07:29:10.500103 8719 raft_consensus.cc:348] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7904b95611f549bca01e22b57d8e6bbd" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 45581 } } peers { permanent_uuid: "f8e6b81932464a7fb592cbbe312ff6a8" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 38431 } } peers { permanent_uuid: "f0883e535e1a4903aec5e84fe7ecdab2" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 34019 } }
I20260810 07:29:10.500346 8719 raft_consensus.cc:374] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:10.500415 8719 raft_consensus.cc:729] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0883e535e1a4903aec5e84fe7ecdab2, State: Initialized, Role: FOLLOWER
I20260810 07:29:10.500847 8212 heartbeater.cc:503] Master 127.12.157.254:41231 was elected leader, sending a full tablet report...
I20260810 07:29:10.500823 8719 consensus_queue.cc:260] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7904b95611f549bca01e22b57d8e6bbd" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 45581 } } peers { permanent_uuid: "f8e6b81932464a7fb592cbbe312ff6a8" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 38431 } } peers { permanent_uuid: "f0883e535e1a4903aec5e84fe7ecdab2" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 34019 } }
I20260810 07:29:10.501066 8718 ts_tablet_manager.cc:1428] T 7275f9f9e7be4e089f4ded93647dec58 P 7904b95611f549bca01e22b57d8e6bbd: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 07:29:10.501750 8719 ts_tablet_manager.cc:1428] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 07:29:10.502079 8441 heartbeater.cc:503] Master 127.12.157.254:41231 was elected leader, sending a full tablet report...
I20260810 07:29:10.503031 8720 raft_consensus.cc:348] T 7275f9f9e7be4e089f4ded93647dec58 P f8e6b81932464a7fb592cbbe312ff6a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7904b95611f549bca01e22b57d8e6bbd" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 45581 } } peers { permanent_uuid: "f8e6b81932464a7fb592cbbe312ff6a8" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 38431 } } peers { permanent_uuid: "f0883e535e1a4903aec5e84fe7ecdab2" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 34019 } }
I20260810 07:29:10.503300 8720 raft_consensus.cc:374] T 7275f9f9e7be4e089f4ded93647dec58 P f8e6b81932464a7fb592cbbe312ff6a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:10.503371 8720 raft_consensus.cc:729] T 7275f9f9e7be4e089f4ded93647dec58 P f8e6b81932464a7fb592cbbe312ff6a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e6b81932464a7fb592cbbe312ff6a8, State: Initialized, Role: FOLLOWER
I20260810 07:29:10.503816 8720 consensus_queue.cc:260] T 7275f9f9e7be4e089f4ded93647dec58 P f8e6b81932464a7fb592cbbe312ff6a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7904b95611f549bca01e22b57d8e6bbd" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 45581 } } peers { permanent_uuid: "f8e6b81932464a7fb592cbbe312ff6a8" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 38431 } } peers { permanent_uuid: "f0883e535e1a4903aec5e84fe7ecdab2" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 34019 } }
I20260810 07:29:10.504588 8689 heartbeater.cc:503] Master 127.12.157.254:41231 was elected leader, sending a full tablet report...
I20260810 07:29:10.504789 8720 ts_tablet_manager.cc:1428] T 7275f9f9e7be4e089f4ded93647dec58 P f8e6b81932464a7fb592cbbe312ff6a8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 07:29:10.613453 8726 raft_consensus.cc:482] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:29:10.613672 8726 raft_consensus.cc:504] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7904b95611f549bca01e22b57d8e6bbd" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 45581 } } peers { permanent_uuid: "f8e6b81932464a7fb592cbbe312ff6a8" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 38431 } } peers { permanent_uuid: "f0883e535e1a4903aec5e84fe7ecdab2" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 34019 } }
I20260810 07:29:10.614786 8726 leader_election.cc:288] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7904b95611f549bca01e22b57d8e6bbd (127.12.157.193:45581), f8e6b81932464a7fb592cbbe312ff6a8 (127.12.157.195:38431)
I20260810 07:29:10.619086 8712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:10.615225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:34561 (local address 127.12.157.193:45581)
0810 07:29:10.615361 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:10.615367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:10.615386 (+ 19us) server_negotiation.cc:407] Connection header received
0810 07:29:10.615454 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:10.615459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:10.615507 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:10.615620 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:10.615999 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.616646 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.617819 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.618424 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.618533 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:10.618609 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:10.618868 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 07:29:10.618960 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 07:29:10.619117 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:10.615525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:47721 (local address 127.12.157.195:38431)
0810 07:29:10.615663 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:10.615668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:10.615856 (+ 188us) server_negotiation.cc:407] Connection header received
0810 07:29:10.615921 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:10.615926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:10.615996 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:10.616057 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:10.616355 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.616901 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.618043 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.618507 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.618624 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:10.618686 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:10.618924 (+ 238us) server_negotiation.cc:299] Negotiation successful
0810 07:29:10.619009 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 07:29:10.619079 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:10.615074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:45581 (local address 127.12.157.194:34561)
0810 07:29:10.615211 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:10.615226 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:10.615322 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:10.615663 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:10.615667 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:10.615690 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:10.615838 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:10.615848 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.616752 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.616755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:10.617678 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:10.617695 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.618549 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.618554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:10.618724 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:10.618753 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 07:29:10.618820 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 07:29:10.618906 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":21}
I20260810 07:29:10.619110 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:10.615339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:38431 (local address 127.12.157.194:47721)
0810 07:29:10.615740 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:10.615754 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:10.615846 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:10.616094 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:10.616096 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:10.616114 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:10.616241 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:10.616248 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.617025 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.617028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:10.617893 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:10.617904 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.618632 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.618637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:10.618773 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:10.618793 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 07:29:10.618892 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 07:29:10.618958 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":37,"thread_start_us":157,"threads_started":1}
I20260810 07:29:10.619833 8132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7275f9f9e7be4e089f4ded93647dec58" candidate_uuid: "f0883e535e1a4903aec5e84fe7ecdab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7904b95611f549bca01e22b57d8e6bbd" is_pre_election: true
I20260810 07:29:10.620083 8616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7275f9f9e7be4e089f4ded93647dec58" candidate_uuid: "f0883e535e1a4903aec5e84fe7ecdab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e6b81932464a7fb592cbbe312ff6a8" is_pre_election: true
I20260810 07:29:10.620218 8132 raft_consensus.cc:2454] T 7275f9f9e7be4e089f4ded93647dec58 P 7904b95611f549bca01e22b57d8e6bbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0883e535e1a4903aec5e84fe7ecdab2 in term 0.
I20260810 07:29:10.620357 8616 raft_consensus.cc:2454] T 7275f9f9e7be4e089f4ded93647dec58 P f8e6b81932464a7fb592cbbe312ff6a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0883e535e1a4903aec5e84fe7ecdab2 in term 0.
I20260810 07:29:10.620790 8243 leader_election.cc:302] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7904b95611f549bca01e22b57d8e6bbd, f0883e535e1a4903aec5e84fe7ecdab2; no voters:
I20260810 07:29:10.621122 8726 raft_consensus.cc:2781] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:29:10.621225 8726 raft_consensus.cc:482] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:29:10.621286 8726 raft_consensus.cc:3037] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:10.622378 8726 raft_consensus.cc:504] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7904b95611f549bca01e22b57d8e6bbd" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 45581 } } peers { permanent_uuid: "f8e6b81932464a7fb592cbbe312ff6a8" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 38431 } } peers { permanent_uuid: "f0883e535e1a4903aec5e84fe7ecdab2" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 34019 } }
I20260810 07:29:10.622737 8726 leader_election.cc:288] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2 [CANDIDATE]: Term 1 election: Requested vote from peers 7904b95611f549bca01e22b57d8e6bbd (127.12.157.193:45581), f8e6b81932464a7fb592cbbe312ff6a8 (127.12.157.195:38431)
I20260810 07:29:10.623239 8616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7275f9f9e7be4e089f4ded93647dec58" candidate_uuid: "f0883e535e1a4903aec5e84fe7ecdab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e6b81932464a7fb592cbbe312ff6a8"
I20260810 07:29:10.623270 8132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7275f9f9e7be4e089f4ded93647dec58" candidate_uuid: "f0883e535e1a4903aec5e84fe7ecdab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7904b95611f549bca01e22b57d8e6bbd"
I20260810 07:29:10.623381 8616 raft_consensus.cc:3037] T 7275f9f9e7be4e089f4ded93647dec58 P f8e6b81932464a7fb592cbbe312ff6a8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:10.623450 8132 raft_consensus.cc:3037] T 7275f9f9e7be4e089f4ded93647dec58 P 7904b95611f549bca01e22b57d8e6bbd [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:10.624769 8616 raft_consensus.cc:2454] T 7275f9f9e7be4e089f4ded93647dec58 P f8e6b81932464a7fb592cbbe312ff6a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0883e535e1a4903aec5e84fe7ecdab2 in term 1.
I20260810 07:29:10.624773 8132 raft_consensus.cc:2454] T 7275f9f9e7be4e089f4ded93647dec58 P 7904b95611f549bca01e22b57d8e6bbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0883e535e1a4903aec5e84fe7ecdab2 in term 1.
I20260810 07:29:10.625108 8243 leader_election.cc:302] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f0883e535e1a4903aec5e84fe7ecdab2, f8e6b81932464a7fb592cbbe312ff6a8; no voters:
I20260810 07:29:10.625326 8726 raft_consensus.cc:2781] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:10.625824 8726 raft_consensus.cc:686] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2 [term 1 LEADER]: Becoming Leader. State: Replica: f0883e535e1a4903aec5e84fe7ecdab2, State: Running, Role: LEADER
I20260810 07:29:10.626423 8726 consensus_queue.cc:237] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7904b95611f549bca01e22b57d8e6bbd" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 45581 } } peers { permanent_uuid: "f8e6b81932464a7fb592cbbe312ff6a8" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 38431 } } peers { permanent_uuid: "f0883e535e1a4903aec5e84fe7ecdab2" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 34019 } }
I20260810 07:29:10.630000 7874 catalog_manager.cc:5495] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0883e535e1a4903aec5e84fe7ecdab2 (127.12.157.194). New cstate: current_term: 1 leader_uuid: "f0883e535e1a4903aec5e84fe7ecdab2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7904b95611f549bca01e22b57d8e6bbd" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 45581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8e6b81932464a7fb592cbbe312ff6a8" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 38431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0883e535e1a4903aec5e84fe7ecdab2" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 34019 } health_report { overall_health: HEALTHY } } }
W20260810 07:29:10.661799 8690 tablet.cc:2369] T 7275f9f9e7be4e089f4ded93647dec58 P f8e6b81932464a7fb592cbbe312ff6a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:29:10.669574 8716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:10.665726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53936 (local address 127.12.157.194:34019)
0810 07:29:10.665850 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:10.665862 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:10.665876 (+ 14us) server_negotiation.cc:407] Connection header received
0810 07:29:10.665989 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:10.665991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:10.666046 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:10.666132 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:29:10.666390 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.666879 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.668085 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.668297 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.668378 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:10.668673 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:29:10.669138 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:29:10.669238 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:10.669333 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 07:29:10.669385 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 07:29:10.676383 8132 raft_consensus.cc:1261] T 7275f9f9e7be4e089f4ded93647dec58 P 7904b95611f549bca01e22b57d8e6bbd [term 1 FOLLOWER]: Refusing update from remote peer f0883e535e1a4903aec5e84fe7ecdab2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:29:10.676374 8616 raft_consensus.cc:1261] T 7275f9f9e7be4e089f4ded93647dec58 P f8e6b81932464a7fb592cbbe312ff6a8 [term 1 FOLLOWER]: Refusing update from remote peer f0883e535e1a4903aec5e84fe7ecdab2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:29:10.677217 8726 consensus_queue.cc:1035] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7904b95611f549bca01e22b57d8e6bbd" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 45581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:29:10.677456 8731 consensus_queue.cc:1035] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e6b81932464a7fb592cbbe312ff6a8" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 38431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:29:10.686321 8737 mvcc.cc:205] Tried to move back new op lower bound from 7316877109961904128 to 7316877109768781824. Current Snapshot: MvccSnapshot[applied={T|T < 7316877109961904128}]
I20260810 07:29:10.696313 7874 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:29:10.696552 7874 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:29:10.699303 7874 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b253743fe8c54fa2b70a77a8e71f0e34: Sending DeleteTablet for 3 replicas of tablet 7275f9f9e7be4e089f4ded93647dec58
I20260810 07:29:10.700213 8324 tablet_service.cc:1514] Processing DeleteTablet for tablet 7275f9f9e7be4e089f4ded93647dec58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:29:10 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:53920
I20260810 07:29:10.700225 8101 tablet_service.cc:1514] Processing DeleteTablet for tablet 7275f9f9e7be4e089f4ded93647dec58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:29:10 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:53030
I20260810 07:29:10.700371 8581 tablet_service.cc:1514] Processing DeleteTablet for tablet 7275f9f9e7be4e089f4ded93647dec58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:29:10 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:46332
I20260810 07:29:10.700894 8754 tablet_replica.cc:330] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2: stopping tablet replica
I20260810 07:29:10.701048 8757 tablet_replica.cc:330] T 7275f9f9e7be4e089f4ded93647dec58 P f8e6b81932464a7fb592cbbe312ff6a8: stopping tablet replica
I20260810 07:29:10.701190 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 7983
I20260810 07:29:10.701277 8754 raft_consensus.cc:2229] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:29:10.702145 8754 raft_consensus.cc:2258] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:29:10.706195 8757 raft_consensus.cc:2229] T 7275f9f9e7be4e089f4ded93647dec58 P f8e6b81932464a7fb592cbbe312ff6a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:29:10.706593 8757 raft_consensus.cc:2258] T 7275f9f9e7be4e089f4ded93647dec58 P f8e6b81932464a7fb592cbbe312ff6a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:29:10.708770 8754 ts_tablet_manager.cc:1905] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:29:10.710170 8757 ts_tablet_manager.cc:1905] T 7275f9f9e7be4e089f4ded93647dec58 P f8e6b81932464a7fb592cbbe312ff6a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:29:10.712603 8754 ts_tablet_manager.cc:1918] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:29:10.712750 8754 log.cc:1196] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-1/wal/wals/7275f9f9e7be4e089f4ded93647dec58
I20260810 07:29:10.713156 8754 ts_tablet_manager.cc:1939] T 7275f9f9e7be4e089f4ded93647dec58 P f0883e535e1a4903aec5e84fe7ecdab2: Deleting consensus metadata
I20260810 07:29:10.713204 8757 ts_tablet_manager.cc:1918] T 7275f9f9e7be4e089f4ded93647dec58 P f8e6b81932464a7fb592cbbe312ff6a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:29:10.713304 8757 log.cc:1196] T 7275f9f9e7be4e089f4ded93647dec58 P f8e6b81932464a7fb592cbbe312ff6a8: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786346694975122-12919-0/minicluster-data/ts-2/wal/wals/7275f9f9e7be4e089f4ded93647dec58
I20260810 07:29:10.713786 8757 ts_tablet_manager.cc:1939] T 7275f9f9e7be4e089f4ded93647dec58 P f8e6b81932464a7fb592cbbe312ff6a8: Deleting consensus metadata
I20260810 07:29:10.714744 7851 catalog_manager.cc:4841] TS f0883e535e1a4903aec5e84fe7ecdab2 (127.12.157.194:34019): tablet 7275f9f9e7be4e089f4ded93647dec58 (table test-table [id=235e573b1d0e430caf79215891a44f08]) successfully deleted
I20260810 07:29:10.715176 7850 catalog_manager.cc:4841] TS f8e6b81932464a7fb592cbbe312ff6a8 (127.12.157.195:38431): tablet 7275f9f9e7be4e089f4ded93647dec58 (table test-table [id=235e573b1d0e430caf79215891a44f08]) successfully deleted
W20260810 07:29:10.718294 7850 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.157.193:45581 (error 108)
I20260810 07:29:10.718568 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 8217
I20260810 07:29:10.725198 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:10.724715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:45581 (local address 127.0.0.1:53032)
0810 07:29:10.724913 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:10.725079 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.193:45581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260810 07:29:10.725411 7850 catalog_manager.cc:4584] TS 7904b95611f549bca01e22b57d8e6bbd (127.12.157.193:45581): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7275f9f9e7be4e089f4ded93647dec58: Network error: Client connection negotiation failed: client connection to 127.12.157.193:45581: connect: Connection refused (error 111)
I20260810 07:29:10.731184 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 8449
I20260810 07:29:10.748601 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 7821
2026-08-10T07:29:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2754 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T07:29:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:10Z Disabled control of system clock
I20260810 07:29:10.789287 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:38045
--webserver_interface=127.12.157.254
--webserver_port=0
--builtin_ntp_servers=127.12.157.212:37757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:38045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 07:29:10.909000 8765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:10.909296 8765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:10.909332 8765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:10.913482 8765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:10.913553 8765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:10.913569 8765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:10.913589 8765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:10.913604 8765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:10.918566 8765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:37757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:38045
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:38045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:10.919705 8765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:10.921145 8765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:10.930008 8771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:10.930027 8775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:10.930051 8772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:10.930414 8765 server_base.cc:1034] running on GCE node
I20260810 07:29:10.931066 8765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:10.932276 8765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:10.933485 8765 hybrid_clock.cc:648] HybridClock initialized: now 1786346950933464 us; error 31 us; skew 500 ppm
I20260810 07:29:10.935469 8765 webserver.cc:460] Webserver started at http://127.12.157.254:42169/ using document root <none> and password file <none>
I20260810 07:29:10.936090 8765 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:10.936189 8765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:10.936440 8765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:10.939133 8765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "c4f4121c7e6e41f9a9989f554bd03f35"
format_stamp: "Formatted at 2026-08-10 07:29:10 on dist-test-slave-ffrd"
server_key: "25c9de8cadfd61943687ef5c8d762559"
server_key_iv: "839b7a457751c292434ca6b12ea12437"
server_key_version: "encryptionkey@0"
I20260810 07:29:10.939733 8765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "c4f4121c7e6e41f9a9989f554bd03f35"
format_stamp: "Formatted at 2026-08-10 07:29:10 on dist-test-slave-ffrd"
server_key: "25c9de8cadfd61943687ef5c8d762559"
server_key_iv: "839b7a457751c292434ca6b12ea12437"
server_key_version: "encryptionkey@0"
I20260810 07:29:10.943579 8765 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 07:29:10.946116 8787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:10.947361 8765 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 07:29:10.947527 8765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "c4f4121c7e6e41f9a9989f554bd03f35"
format_stamp: "Formatted at 2026-08-10 07:29:10 on dist-test-slave-ffrd"
server_key: "25c9de8cadfd61943687ef5c8d762559"
server_key_iv: "839b7a457751c292434ca6b12ea12437"
server_key_version: "encryptionkey@0"
I20260810 07:29:10.947649 8765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:10.960204 8765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:10.960906 8765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:10.961141 8765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:10.968695 8765 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:38045
I20260810 07:29:10.968698 8870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:38045 every 8 connection(s)
I20260810 07:29:10.969729 8765 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:10.972885 8872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:10.976434 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 8765
I20260810 07:29:10.976598 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:29:10.976855 12919 external_mini_cluster.cc:1428] Setting key 0fe3f4a687d74bbe1cadc576a75c0f73
I20260810 07:29:10.979084 8872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4f4121c7e6e41f9a9989f554bd03f35: Bootstrap starting.
I20260810 07:29:10.981897 8872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4f4121c7e6e41f9a9989f554bd03f35: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:10.982792 8872 log.cc:824] T 00000000000000000000000000000000 P c4f4121c7e6e41f9a9989f554bd03f35: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:10.985106 8872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4f4121c7e6e41f9a9989f554bd03f35: No bootstrap required, opened a new log
I20260810 07:29:10.987883 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:10.978400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57744 (local address 127.12.157.254:38045)
0810 07:29:10.979134 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:10.979159 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:10.979202 (+ 43us) server_negotiation.cc:407] Connection header received
0810 07:29:10.980032 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:10.980055 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:10.980587 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:10.980975 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:10.981452 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.982953 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.984057 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:10.984748 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:10.984903 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:10.985824 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:10.985830 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:10.985847 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:10.986347 (+ 500us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:10.986416 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:10.986442 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:10.986545 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:10.986650 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 07:29:10.986926 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":124,"threads_started":1}
I20260810 07:29:10.989791 8872 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4f4121c7e6e41f9a9989f554bd03f35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f4121c7e6e41f9a9989f554bd03f35" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 38045 } }
I20260810 07:29:10.990067 8872 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4f4121c7e6e41f9a9989f554bd03f35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:10.990151 8872 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4f4121c7e6e41f9a9989f554bd03f35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f4121c7e6e41f9a9989f554bd03f35, State: Initialized, Role: FOLLOWER
I20260810 07:29:10.990628 8872 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4f4121c7e6e41f9a9989f554bd03f35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f4121c7e6e41f9a9989f554bd03f35" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 38045 } }
I20260810 07:29:10.990801 8872 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4f4121c7e6e41f9a9989f554bd03f35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:29:10.990892 8872 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4f4121c7e6e41f9a9989f554bd03f35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:29:10.990995 8872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4f4121c7e6e41f9a9989f554bd03f35 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:10.992198 8872 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4f4121c7e6e41f9a9989f554bd03f35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f4121c7e6e41f9a9989f554bd03f35" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 38045 } }
I20260810 07:29:10.992568 8872 leader_election.cc:302] T 00000000000000000000000000000000 P c4f4121c7e6e41f9a9989f554bd03f35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4f4121c7e6e41f9a9989f554bd03f35; no voters:
I20260810 07:29:10.992918 8872 leader_election.cc:288] T 00000000000000000000000000000000 P c4f4121c7e6e41f9a9989f554bd03f35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:29:10.993096 8880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4f4121c7e6e41f9a9989f554bd03f35 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:10.993340 8880 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4f4121c7e6e41f9a9989f554bd03f35 [term 1 LEADER]: Becoming Leader. State: Replica: c4f4121c7e6e41f9a9989f554bd03f35, State: Running, Role: LEADER
I20260810 07:29:10.993654 8880 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4f4121c7e6e41f9a9989f554bd03f35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f4121c7e6e41f9a9989f554bd03f35" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 38045 } }
I20260810 07:29:10.994194 8872 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4f4121c7e6e41f9a9989f554bd03f35 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:10.995271 8881 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4f4121c7e6e41f9a9989f554bd03f35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4f4121c7e6e41f9a9989f554bd03f35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f4121c7e6e41f9a9989f554bd03f35" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 38045 } } }
I20260810 07:29:10.995396 8881 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4f4121c7e6e41f9a9989f554bd03f35 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:10.995788 8895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:10.995788 8886 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4f4121c7e6e41f9a9989f554bd03f35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4f4121c7e6e41f9a9989f554bd03f35. Latest consensus state: current_term: 1 leader_uuid: "c4f4121c7e6e41f9a9989f554bd03f35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4f4121c7e6e41f9a9989f554bd03f35" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 38045 } } }
I20260810 07:29:10.996474 8886 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4f4121c7e6e41f9a9989f554bd03f35 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:10.998713 8895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:11.004675 8895 catalog_manager.cc:1347] Generated new cluster ID: effea8aef7ec4e10baf2d465c0a350a8
I20260810 07:29:11.004756 8895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:11.009512 8895 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:11.010504 8895 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:11.017655 8895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4f4121c7e6e41f9a9989f554bd03f35: Generated new TSK 0
I20260810 07:29:11.018458 8895 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:29:11.029654 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:0
--local_ip_for_outbound_sockets=127.12.157.193
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:38045
--builtin_ntp_servers=127.12.157.212:37757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 07:29:11.150552 8907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:11.150890 8907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:11.150930 8907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:11.155151 8907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:11.155217 8907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:11.155278 8907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:29:11.160312 8907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:37757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:38045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:11.161485 8907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:11.162907 8907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:11.172932 8916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:11.172983 8926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:11.173030 8907 server_base.cc:1034] running on GCE node
W20260810 07:29:11.172983 8919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:11.173650 8907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:11.174335 8907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:11.175546 8907 hybrid_clock.cc:648] HybridClock initialized: now 1786346951175527 us; error 28 us; skew 500 ppm
I20260810 07:29:11.177644 8907 webserver.cc:460] Webserver started at http://127.12.157.193:44343/ using document root <none> and password file <none>
I20260810 07:29:11.178256 8907 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:11.178344 8907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:11.178586 8907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:11.180924 8907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-0/data/instance:
uuid: "76d191d270f7467dbdfbfa54b946a937"
format_stamp: "Formatted at 2026-08-10 07:29:11 on dist-test-slave-ffrd"
server_key: "f1da74359cc60b4eeada7f676eccaf2d"
server_key_iv: "a625ed62232430c3e157e05711881817"
server_key_version: "encryptionkey@0"
I20260810 07:29:11.181622 8907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance:
uuid: "76d191d270f7467dbdfbfa54b946a937"
format_stamp: "Formatted at 2026-08-10 07:29:11 on dist-test-slave-ffrd"
server_key: "f1da74359cc60b4eeada7f676eccaf2d"
server_key_iv: "a625ed62232430c3e157e05711881817"
server_key_version: "encryptionkey@0"
I20260810 07:29:11.185793 8907 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 07:29:11.188587 8935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:11.189977 8907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:29:11.190128 8907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "76d191d270f7467dbdfbfa54b946a937"
format_stamp: "Formatted at 2026-08-10 07:29:11 on dist-test-slave-ffrd"
server_key: "f1da74359cc60b4eeada7f676eccaf2d"
server_key_iv: "a625ed62232430c3e157e05711881817"
server_key_version: "encryptionkey@0"
I20260810 07:29:11.190256 8907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:11.206719 8907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:11.207612 8907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:11.207842 8907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:11.208513 8907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:29:11.209636 8907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:29:11.209712 8907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:11.209801 8907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:29:11.209846 8907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:11.219488 8907 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:39897
I20260810 07:29:11.219583 9121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:39897 every 8 connection(s)
I20260810 07:29:11.220510 8907 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:29:11.228240 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 8907
I20260810 07:29:11.228417 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance
I20260810 07:29:11.228703 12919 external_mini_cluster.cc:1428] Setting key dbf05e1fb6ec2164c0f0554d44e68507
I20260810 07:29:11.230909 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:0
--local_ip_for_outbound_sockets=127.12.157.194
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:38045
--builtin_ntp_servers=127.12.157.212:37757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 07:29:11.232023 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:11.222564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:35903 (local address 127.12.157.254:38045)
0810 07:29:11.222718 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:11.222722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:11.223943 (+ 1221us) server_negotiation.cc:407] Connection header received
0810 07:29:11.226205 (+ 2262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:11.226209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:11.226263 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:11.226344 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:11.227525 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.228126 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.229742 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.229975 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.230096 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:11.231482 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:11.231486 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:11.231488 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:11.231501 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:11.231522 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:11.231532 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:11.231624 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:11.231802 (+ 178us) server_negotiation.cc:299] Negotiation successful
0810 07:29:11.231876 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 07:29:11.233119 9128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:11.222958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:38045 (local address 127.12.157.193:35903)
0810 07:29:11.223710 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:11.223755 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:11.225932 (+ 2177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:11.226542 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:11.226549 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:11.226640 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:11.227204 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:11.227218 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.228520 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.228524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:11.229599 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:11.229611 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.230158 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.230163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:11.230342 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:11.231100 (+ 758us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:29:11.231102 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:11.231169 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:29:11.231257 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:11.231629 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:11.231642 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:11.231649 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 07:29:11.231811 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 07:29:11.232144 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":124,"threads_started":1}
I20260810 07:29:11.234542 9122 heartbeater.cc:347] Connected to a master server at 127.12.157.254:38045
I20260810 07:29:11.234835 9122 heartbeater.cc:464] Registering TS with master...
I20260810 07:29:11.235445 9122 heartbeater.cc:511] Master 127.12.157.254:38045 requested a full tablet report, sending...
I20260810 07:29:11.237226 8811 ts_manager.cc:194] Registered new tserver with Master: 76d191d270f7467dbdfbfa54b946a937 (127.12.157.193:39897)
I20260810 07:29:11.238560 8811 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.157.193:35903
W20260810 07:29:11.356923 9131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:11.357280 9131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:11.357362 9131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:11.361583 9131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:11.361657 9131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:11.361742 9131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:29:11.366672 9131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:37757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:38045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:11.367774 9131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:11.369382 9131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:11.378921 9140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:11.378943 9136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:11.379019 9137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:11.379302 9131 server_base.cc:1034] running on GCE node
I20260810 07:29:11.379972 9131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:11.380695 9131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:11.381920 9131 hybrid_clock.cc:648] HybridClock initialized: now 1786346951381880 us; error 51 us; skew 500 ppm
I20260810 07:29:11.384156 9131 webserver.cc:460] Webserver started at http://127.12.157.194:33405/ using document root <none> and password file <none>
I20260810 07:29:11.384829 9131 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:11.384938 9131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:11.385231 9131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:11.387569 9131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-1/data/instance:
uuid: "8355a4162d9b4b8e98320acb64e33623"
format_stamp: "Formatted at 2026-08-10 07:29:11 on dist-test-slave-ffrd"
server_key: "d66861272bdfdc9da40466965d11ad65"
server_key_iv: "5fcf211a9e51ff6edd2401a5d251837f"
server_key_version: "encryptionkey@0"
I20260810 07:29:11.388259 9131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance:
uuid: "8355a4162d9b4b8e98320acb64e33623"
format_stamp: "Formatted at 2026-08-10 07:29:11 on dist-test-slave-ffrd"
server_key: "d66861272bdfdc9da40466965d11ad65"
server_key_iv: "5fcf211a9e51ff6edd2401a5d251837f"
server_key_version: "encryptionkey@0"
I20260810 07:29:11.392421 9131 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 07:29:11.395292 9148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:11.396735 9131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 07:29:11.396878 9131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "8355a4162d9b4b8e98320acb64e33623"
format_stamp: "Formatted at 2026-08-10 07:29:11 on dist-test-slave-ffrd"
server_key: "d66861272bdfdc9da40466965d11ad65"
server_key_iv: "5fcf211a9e51ff6edd2401a5d251837f"
server_key_version: "encryptionkey@0"
I20260810 07:29:11.397066 9131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:11.427155 9131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:11.427950 9131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:11.428172 9131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:11.428809 9131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:29:11.429831 9131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:29:11.429919 9131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:11.429993 9131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:29:11.430045 9131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:11.439829 9131 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:38969
I20260810 07:29:11.439867 9336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:38969 every 8 connection(s)
I20260810 07:29:11.440819 9131 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:29:11.450538 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:11.442802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:41231 (local address 127.12.157.254:38045)
0810 07:29:11.442951 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:11.442955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:11.444082 (+ 1127us) server_negotiation.cc:407] Connection header received
0810 07:29:11.445626 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:11.445629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:11.445695 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:11.445800 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:11.446720 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.447261 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.448408 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.448640 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.448755 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:11.449957 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:11.449960 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:11.449962 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:11.449975 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:11.449994 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:11.450002 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:11.450084 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:11.450289 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 07:29:11.450349 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 07:29:11.450534 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 9131
I20260810 07:29:11.450700 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance
I20260810 07:29:11.451025 12919 external_mini_cluster.cc:1428] Setting key fc424b0d01f5f6b78e2e4cbc773b874f
I20260810 07:29:11.451799 9340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:11.443210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:38045 (local address 127.12.157.194:41231)
0810 07:29:11.443908 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:11.443959 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:11.445394 (+ 1435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:11.445990 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:11.445997 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:11.446081 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:11.446577 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:11.446589 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.447402 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.447405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:11.448278 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:11.448288 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.448802 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.448806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:11.449038 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:11.449663 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:29:11.449664 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:11.449704 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:29:11.449767 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:11.450085 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:11.450097 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:11.450106 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 07:29:11.450247 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 07:29:11.450657 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":130,"threads_started":1}
I20260810 07:29:11.453094 9338 heartbeater.cc:347] Connected to a master server at 127.12.157.254:38045
I20260810 07:29:11.453354 9338 heartbeater.cc:464] Registering TS with master...
I20260810 07:29:11.453639 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:0
--local_ip_for_outbound_sockets=127.12.157.195
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:38045
--builtin_ntp_servers=127.12.157.212:37757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 07:29:11.453958 9338 heartbeater.cc:511] Master 127.12.157.254:38045 requested a full tablet report, sending...
I20260810 07:29:11.455056 8811 ts_manager.cc:194] Registered new tserver with Master: 8355a4162d9b4b8e98320acb64e33623 (127.12.157.194:38969)
I20260810 07:29:11.455825 8811 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.157.194:41231
W20260810 07:29:11.578012 9341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:11.578285 9341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:11.578368 9341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:11.582473 9341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:11.582526 9341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:11.582582 9341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:29:11.587260 9341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:37757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:38045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:11.588299 9341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:11.589707 9341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:11.599664 9354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:11.599666 9348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:11.599666 9350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:11.600009 9341 server_base.cc:1034] running on GCE node
I20260810 07:29:11.600706 9341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:11.601670 9341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:11.602892 9341 hybrid_clock.cc:648] HybridClock initialized: now 1786346951602881 us; error 47 us; skew 500 ppm
I20260810 07:29:11.605183 9341 webserver.cc:460] Webserver started at http://127.12.157.195:42415/ using document root <none> and password file <none>
I20260810 07:29:11.605792 9341 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:11.605891 9341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:11.606140 9341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:11.608275 9341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-2/data/instance:
uuid: "fa35dc091bd34f1a9085c6a7793fbe28"
format_stamp: "Formatted at 2026-08-10 07:29:11 on dist-test-slave-ffrd"
server_key: "c1acd097e12da715c8bad26bd049ad88"
server_key_iv: "74336d37afd8283f1fb707d534837280"
server_key_version: "encryptionkey@0"
I20260810 07:29:11.608867 9341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance:
uuid: "fa35dc091bd34f1a9085c6a7793fbe28"
format_stamp: "Formatted at 2026-08-10 07:29:11 on dist-test-slave-ffrd"
server_key: "c1acd097e12da715c8bad26bd049ad88"
server_key_iv: "74336d37afd8283f1fb707d534837280"
server_key_version: "encryptionkey@0"
I20260810 07:29:11.612658 9341 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 07:29:11.615294 9364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:11.616587 9341 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 07:29:11.616729 9341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "fa35dc091bd34f1a9085c6a7793fbe28"
format_stamp: "Formatted at 2026-08-10 07:29:11 on dist-test-slave-ffrd"
server_key: "c1acd097e12da715c8bad26bd049ad88"
server_key_iv: "74336d37afd8283f1fb707d534837280"
server_key_version: "encryptionkey@0"
I20260810 07:29:11.616847 9341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:11.673000 9341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:11.673803 9341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:11.674041 9341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:11.674708 9341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:29:11.675760 9341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:29:11.675834 9341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:11.675912 9341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:29:11.675969 9341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:11.685827 9341 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:34353
I20260810 07:29:11.685859 9557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:34353 every 8 connection(s)
I20260810 07:29:11.686770 9341 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:29:11.693039 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 9341
I20260810 07:29:11.693158 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance
I20260810 07:29:11.693431 12919 external_mini_cluster.cc:1428] Setting key eb86fabdcb078d3fe290f841fa6387a2
I20260810 07:29:11.697553 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:11.689096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:49207 (local address 127.12.157.254:38045)
0810 07:29:11.689246 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:11.689249 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:11.690384 (+ 1135us) server_negotiation.cc:407] Connection header received
0810 07:29:11.691940 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:11.691944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:11.691994 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:11.692062 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:11.693045 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.693832 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.695660 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.695889 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.695984 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:11.697148 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:11.697151 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:11.697153 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:11.697166 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:11.697187 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:11.697195 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:11.697268 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:11.697383 (+ 115us) server_negotiation.cc:299] Negotiation successful
0810 07:29:11.697442 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 07:29:11.698453 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:11.689519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:38045 (local address 127.12.157.195:49207)
0810 07:29:11.690213 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:11.690258 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:11.691643 (+ 1385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:11.692228 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:11.692235 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:11.692317 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:11.692829 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:11.692868 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.693973 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.693978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:11.695519 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:11.695534 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.696017 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.696020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:11.696163 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:11.696858 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:29:11.696859 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:11.696907 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:29:11.696999 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:11.697268 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:11.697272 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:11.697276 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:29:11.697363 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 07:29:11.697595 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":114,"threads_started":1}
I20260810 07:29:11.699637 9558 heartbeater.cc:347] Connected to a master server at 127.12.157.254:38045
I20260810 07:29:11.699903 9558 heartbeater.cc:464] Registering TS with master...
I20260810 07:29:11.700534 9558 heartbeater.cc:511] Master 127.12.157.254:38045 requested a full tablet report, sending...
I20260810 07:29:11.701753 8811 ts_manager.cc:194] Registered new tserver with Master: fa35dc091bd34f1a9085c6a7793fbe28 (127.12.157.195:34353)
I20260810 07:29:11.702364 8811 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.157.195:49207
I20260810 07:29:11.707984 12919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:29:11.714509 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:11.710082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57746 (local address 127.12.157.254:38045)
0810 07:29:11.710231 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:11.710235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:11.710462 (+ 227us) server_negotiation.cc:407] Connection header received
0810 07:29:11.710540 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:11.710543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:11.710587 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:11.710659 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:11.710946 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.711686 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.712677 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.713027 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.713115 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:11.714046 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:11.714049 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:11.714051 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:11.714063 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:11.714082 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:11.714090 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:11.714153 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:11.714266 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 07:29:11.714322 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 07:29:11.717638 8811 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:29:11.720837 8811 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:29:11.736737 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:11.730361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:38969 (local address 127.0.0.1:37666)
0810 07:29:11.731213 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:11.731279 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:11.731431 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:11.732299 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:11.732306 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:11.732331 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:11.732492 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:11.732500 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.734016 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.734022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:11.735272 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:11.735282 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.736085 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.736089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:11.736219 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:11.736284 (+ 65us) client_negotiation.cc:769] Sending connection context
0810 07:29:11.736405 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 07:29:11.736539 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":25,"spinlock_wait_cycles":13952,"thread_start_us":101,"threads_started":1}
I20260810 07:29:11.736799 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:11.730709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37666 (local address 127.12.157.194:38969)
0810 07:29:11.731769 (+ 1060us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:11.731782 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:11.731803 (+ 21us) server_negotiation.cc:407] Connection header received
0810 07:29:11.731883 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:11.731888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:11.732114 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:11.732299 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:11.732720 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.733904 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.735400 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.735965 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.736091 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:11.736279 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:11.736450 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 07:29:11.736554 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":944,"thread_start_us":159,"threads_started":1}
I20260810 07:29:11.737531 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:11.730382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:39897 (local address 127.0.0.1:58128)
0810 07:29:11.731213 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:11.731279 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:11.731431 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:11.732739 (+ 1308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:11.732743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:11.732764 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:11.732867 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:11.732873 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.734456 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.734459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:11.735864 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:11.735876 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.737152 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.737155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:11.737219 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:11.737236 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 07:29:11.737329 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 07:29:11.737394 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":33,"spinlock_wait_cycles":22272,"thread_start_us":96,"threads_started":1}
I20260810 07:29:11.737960 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:11.730663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58128 (local address 127.12.157.193:39897)
0810 07:29:11.731756 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:11.731766 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:11.731797 (+ 31us) server_negotiation.cc:407] Connection header received
0810 07:29:11.731883 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:11.731888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:11.732114 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:11.732724 (+ 610us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:11.733022 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.734306 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.736006 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.736760 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.736900 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:11.737094 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:11.737720 (+ 626us) server_negotiation.cc:299] Negotiation successful
0810 07:29:11.737828 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":967,"thread_start_us":113,"threads_started":1}
I20260810 07:29:11.738296 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:11.731134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:34353 (local address 127.0.0.1:36050)
0810 07:29:11.732256 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:11.732272 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:11.732399 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:11.733209 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:11.733212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:11.733231 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:11.733335 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:11.733340 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.734826 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.734830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:11.736131 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:11.736140 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.737334 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.737338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:11.737998 (+ 660us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:11.738023 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 07:29:11.738102 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 07:29:11.738169 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1059,"mutex_wait_us":43,"spinlock_wait_cycles":896,"thread_start_us":98,"threads_started":1}
I20260810 07:29:11.738788 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:11.731943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36050 (local address 127.12.157.195:34353)
0810 07:29:11.732641 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:11.732657 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:11.732683 (+ 26us) server_negotiation.cc:407] Connection header received
0810 07:29:11.732789 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:11.732794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:11.733051 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:11.733222 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:11.733464 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.734675 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.736276 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.737197 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.737352 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:11.737539 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:11.738501 (+ 962us) server_negotiation.cc:299] Negotiation successful
0810 07:29:11.738610 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":180,"threads_started":1}
I20260810 07:29:11.740521 9019 tablet_service.cc:1467] Processing CreateTablet for tablet 6c3e2d733024479c920114cf844b3d2d (DEFAULT_TABLE table=test-table [id=cbbc773376cc4baab716918f23fc962f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:29:11.740696 9226 tablet_service.cc:1467] Processing CreateTablet for tablet 6c3e2d733024479c920114cf844b3d2d (DEFAULT_TABLE table=test-table [id=cbbc773376cc4baab716918f23fc962f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:29:11.741641 9019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c3e2d733024479c920114cf844b3d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:11.741514 9446 tablet_service.cc:1467] Processing CreateTablet for tablet 6c3e2d733024479c920114cf844b3d2d (DEFAULT_TABLE table=test-table [id=cbbc773376cc4baab716918f23fc962f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:29:11.742054 9226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c3e2d733024479c920114cf844b3d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:11.742568 9446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c3e2d733024479c920114cf844b3d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:11.748409 9585 tablet_bootstrap.cc:492] T 6c3e2d733024479c920114cf844b3d2d P 76d191d270f7467dbdfbfa54b946a937: Bootstrap starting.
I20260810 07:29:11.750200 9586 tablet_bootstrap.cc:492] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623: Bootstrap starting.
I20260810 07:29:11.750795 9589 tablet_bootstrap.cc:492] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28: Bootstrap starting.
I20260810 07:29:11.750996 9585 tablet_bootstrap.cc:654] T 6c3e2d733024479c920114cf844b3d2d P 76d191d270f7467dbdfbfa54b946a937: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:11.751945 9585 log.cc:824] T 6c3e2d733024479c920114cf844b3d2d P 76d191d270f7467dbdfbfa54b946a937: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:11.754271 9585 tablet_bootstrap.cc:492] T 6c3e2d733024479c920114cf844b3d2d P 76d191d270f7467dbdfbfa54b946a937: No bootstrap required, opened a new log
I20260810 07:29:11.754454 9586 tablet_bootstrap.cc:654] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:11.754494 9585 ts_tablet_manager.cc:1397] T 6c3e2d733024479c920114cf844b3d2d P 76d191d270f7467dbdfbfa54b946a937: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 07:29:11.755347 9586 log.cc:824] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:11.756796 9589 tablet_bootstrap.cc:654] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:11.757529 9586 tablet_bootstrap.cc:492] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623: No bootstrap required, opened a new log
I20260810 07:29:11.757769 9586 ts_tablet_manager.cc:1397] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20260810 07:29:11.757762 9589 log.cc:824] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:11.759222 9585 raft_consensus.cc:348] T 6c3e2d733024479c920114cf844b3d2d P 76d191d270f7467dbdfbfa54b946a937 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8355a4162d9b4b8e98320acb64e33623" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38969 } } peers { permanent_uuid: "76d191d270f7467dbdfbfa54b946a937" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39897 } } peers { permanent_uuid: "fa35dc091bd34f1a9085c6a7793fbe28" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34353 } }
I20260810 07:29:11.759541 9585 raft_consensus.cc:374] T 6c3e2d733024479c920114cf844b3d2d P 76d191d270f7467dbdfbfa54b946a937 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:11.759598 9585 raft_consensus.cc:729] T 6c3e2d733024479c920114cf844b3d2d P 76d191d270f7467dbdfbfa54b946a937 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d191d270f7467dbdfbfa54b946a937, State: Initialized, Role: FOLLOWER
I20260810 07:29:11.760052 9589 tablet_bootstrap.cc:492] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28: No bootstrap required, opened a new log
I20260810 07:29:11.760133 9585 consensus_queue.cc:260] T 6c3e2d733024479c920114cf844b3d2d P 76d191d270f7467dbdfbfa54b946a937 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8355a4162d9b4b8e98320acb64e33623" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38969 } } peers { permanent_uuid: "76d191d270f7467dbdfbfa54b946a937" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39897 } } peers { permanent_uuid: "fa35dc091bd34f1a9085c6a7793fbe28" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34353 } }
I20260810 07:29:11.760236 9589 ts_tablet_manager.cc:1397] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260810 07:29:11.761334 9585 ts_tablet_manager.cc:1428] T 6c3e2d733024479c920114cf844b3d2d P 76d191d270f7467dbdfbfa54b946a937: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260810 07:29:11.761473 9122 heartbeater.cc:503] Master 127.12.157.254:38045 was elected leader, sending a full tablet report...
I20260810 07:29:11.762543 9586 raft_consensus.cc:348] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8355a4162d9b4b8e98320acb64e33623" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38969 } } peers { permanent_uuid: "76d191d270f7467dbdfbfa54b946a937" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39897 } } peers { permanent_uuid: "fa35dc091bd34f1a9085c6a7793fbe28" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34353 } }
I20260810 07:29:11.762851 9586 raft_consensus.cc:374] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:11.762923 9586 raft_consensus.cc:729] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8355a4162d9b4b8e98320acb64e33623, State: Initialized, Role: FOLLOWER
I20260810 07:29:11.763450 9586 consensus_queue.cc:260] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8355a4162d9b4b8e98320acb64e33623" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38969 } } peers { permanent_uuid: "76d191d270f7467dbdfbfa54b946a937" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39897 } } peers { permanent_uuid: "fa35dc091bd34f1a9085c6a7793fbe28" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34353 } }
I20260810 07:29:11.764178 9338 heartbeater.cc:503] Master 127.12.157.254:38045 was elected leader, sending a full tablet report...
I20260810 07:29:11.764077 9589 raft_consensus.cc:348] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8355a4162d9b4b8e98320acb64e33623" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38969 } } peers { permanent_uuid: "76d191d270f7467dbdfbfa54b946a937" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39897 } } peers { permanent_uuid: "fa35dc091bd34f1a9085c6a7793fbe28" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34353 } }
I20260810 07:29:11.764333 9589 raft_consensus.cc:374] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:11.764381 9589 raft_consensus.cc:729] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa35dc091bd34f1a9085c6a7793fbe28, State: Initialized, Role: FOLLOWER
I20260810 07:29:11.764623 9586 ts_tablet_manager.cc:1428] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 07:29:11.764812 9589 consensus_queue.cc:260] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8355a4162d9b4b8e98320acb64e33623" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38969 } } peers { permanent_uuid: "76d191d270f7467dbdfbfa54b946a937" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39897 } } peers { permanent_uuid: "fa35dc091bd34f1a9085c6a7793fbe28" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34353 } }
I20260810 07:29:11.765555 9558 heartbeater.cc:503] Master 127.12.157.254:38045 was elected leader, sending a full tablet report...
I20260810 07:29:11.765857 9589 ts_tablet_manager.cc:1428] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 07:29:11.916368 9598 raft_consensus.cc:482] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:29:11.916599 9598 raft_consensus.cc:504] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8355a4162d9b4b8e98320acb64e33623" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38969 } } peers { permanent_uuid: "76d191d270f7467dbdfbfa54b946a937" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39897 } } peers { permanent_uuid: "fa35dc091bd34f1a9085c6a7793fbe28" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34353 } }
I20260810 07:29:11.917829 9598 leader_election.cc:288] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8355a4162d9b4b8e98320acb64e33623 (127.12.157.194:38969), 76d191d270f7467dbdfbfa54b946a937 (127.12.157.193:39897)
I20260810 07:29:11.918205 9597 raft_consensus.cc:482] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:29:11.918416 9597 raft_consensus.cc:504] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8355a4162d9b4b8e98320acb64e33623" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38969 } } peers { permanent_uuid: "76d191d270f7467dbdfbfa54b946a937" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39897 } } peers { permanent_uuid: "fa35dc091bd34f1a9085c6a7793fbe28" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34353 } }
I20260810 07:29:11.919639 9597 leader_election.cc:288] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d191d270f7467dbdfbfa54b946a937 (127.12.157.193:39897), fa35dc091bd34f1a9085c6a7793fbe28 (127.12.157.195:34353)
I20260810 07:29:11.922816 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:11.918114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:39897 (local address 127.12.157.195:44607)
0810 07:29:11.918287 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:11.918306 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:11.918420 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:11.918803 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:11.918806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:11.918831 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:11.918963 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:11.918974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.920031 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.920034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:11.921335 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:11.921346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.922396 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.922400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:11.922504 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:11.922527 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 07:29:11.922598 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 07:29:11.922686 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 07:29:11.923355 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:11.918383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:44607 (local address 127.12.157.193:39897)
0810 07:29:11.918527 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:11.918532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:11.918549 (+ 17us) server_negotiation.cc:407] Connection header received
0810 07:29:11.918611 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:11.918614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:11.918675 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:11.918778 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:11.919112 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.919871 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.921481 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.922161 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.922268 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:11.922320 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:11.923135 (+ 815us) server_negotiation.cc:299] Negotiation successful
0810 07:29:11.923226 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":60}
I20260810 07:29:11.924008 9054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3e2d733024479c920114cf844b3d2d" candidate_uuid: "fa35dc091bd34f1a9085c6a7793fbe28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d191d270f7467dbdfbfa54b946a937" is_pre_election: true
I20260810 07:29:11.924311 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:11.918280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:44889 (local address 127.12.157.194:38969)
0810 07:29:11.918400 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:11.918404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:11.919152 (+ 748us) server_negotiation.cc:407] Connection header received
0810 07:29:11.919355 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:11.919358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:11.919409 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:11.919457 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:11.919944 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.920685 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.922154 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.923380 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.923505 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:11.923559 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:11.924091 (+ 532us) server_negotiation.cc:299] Negotiation successful
0810 07:29:11.924163 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":568,"server-negotiator.queue_time_us":44}
I20260810 07:29:11.924361 9054 raft_consensus.cc:2454] T 6c3e2d733024479c920114cf844b3d2d P 76d191d270f7467dbdfbfa54b946a937 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa35dc091bd34f1a9085c6a7793fbe28 in term 0.
I20260810 07:29:11.924723 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:11.918122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:38969 (local address 127.12.157.195:44889)
0810 07:29:11.919059 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:11.919076 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:11.919221 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:11.919582 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:11.919585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:11.919601 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:11.919790 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:11.919798 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.920856 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.920859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:11.922003 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:11.922014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.923857 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.923860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:11.923960 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:11.923984 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 07:29:11.924506 (+ 522us) client_negotiation.cc:240] Negotiation successful
0810 07:29:11.924577 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"mutex_wait_us":432,"thread_start_us":147,"threads_started":1}
I20260810 07:29:11.924989 9369 leader_election.cc:302] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d191d270f7467dbdfbfa54b946a937, fa35dc091bd34f1a9085c6a7793fbe28; no voters:
I20260810 07:29:11.925184 9340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:11.919997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:39897 (local address 127.12.157.194:48147)
0810 07:29:11.920478 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:11.920494 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:11.920580 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:11.921076 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:11.921079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:11.921102 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:11.921237 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:11.921247 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.922421 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.922425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:11.923678 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:11.923688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.924786 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.924790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:11.924898 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:11.924921 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 07:29:11.925003 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 07:29:11.925071 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":25}
I20260810 07:29:11.925330 9598 raft_consensus.cc:2781] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:29:11.925431 9598 raft_consensus.cc:482] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:29:11.925393 9256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3e2d733024479c920114cf844b3d2d" candidate_uuid: "fa35dc091bd34f1a9085c6a7793fbe28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8355a4162d9b4b8e98320acb64e33623" is_pre_election: true
I20260810 07:29:11.925499 9598 raft_consensus.cc:3037] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:11.925597 9256 raft_consensus.cc:2454] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa35dc091bd34f1a9085c6a7793fbe28 in term 0.
I20260810 07:29:11.925724 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:11.920415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:48147 (local address 127.12.157.193:39897)
0810 07:29:11.920798 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:11.920807 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:11.920820 (+ 13us) server_negotiation.cc:407] Connection header received
0810 07:29:11.920862 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:11.920865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:11.920916 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:11.921352 (+ 436us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:11.921500 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.922307 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.923775 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.924633 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.925412 (+ 779us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:11.925474 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:11.925512 (+ 38us) server_negotiation.cc:299] Negotiation successful
0810 07:29:11.925587 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":311,"thread_start_us":122,"threads_started":1}
I20260810 07:29:11.926052 9054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3e2d733024479c920114cf844b3d2d" candidate_uuid: "8355a4162d9b4b8e98320acb64e33623" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d191d270f7467dbdfbfa54b946a937" is_pre_election: true
I20260810 07:29:11.926194 9054 raft_consensus.cc:2454] T 6c3e2d733024479c920114cf844b3d2d P 76d191d270f7467dbdfbfa54b946a937 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8355a4162d9b4b8e98320acb64e33623 in term 0.
I20260810 07:29:11.926342 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:11.921114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:58453 (local address 127.12.157.195:34353)
0810 07:29:11.921226 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:11.921231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:11.921618 (+ 387us) server_negotiation.cc:407] Connection header received
0810 07:29:11.921788 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:11.921791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:11.921835 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:11.921879 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:11.922246 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.922978 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.924315 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.924873 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.925026 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:11.925068 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:11.926152 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0810 07:29:11.926244 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 07:29:11.926666 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:11.920834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:34353 (local address 127.12.157.194:58453)
0810 07:29:11.921542 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:11.921555 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:11.921678 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:11.922009 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:11.922012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:11.922028 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:11.922137 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:11.922142 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.923116 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.923119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:11.924189 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:11.924197 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.925120 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.925123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:11.925995 (+ 872us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:11.926020 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 07:29:11.926454 (+ 434us) client_negotiation.cc:240] Negotiation successful
0810 07:29:11.926536 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":391,"threads_started":1}
I20260810 07:29:11.926707 9156 leader_election.cc:302] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d191d270f7467dbdfbfa54b946a937, 8355a4162d9b4b8e98320acb64e33623; no voters:
I20260810 07:29:11.926929 9598 raft_consensus.cc:504] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8355a4162d9b4b8e98320acb64e33623" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38969 } } peers { permanent_uuid: "76d191d270f7467dbdfbfa54b946a937" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39897 } } peers { permanent_uuid: "fa35dc091bd34f1a9085c6a7793fbe28" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34353 } }
I20260810 07:29:11.926983 9597 raft_consensus.cc:2781] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:29:11.927078 9597 raft_consensus.cc:482] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:29:11.927129 9597 raft_consensus.cc:3037] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:11.927258 9475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3e2d733024479c920114cf844b3d2d" candidate_uuid: "8355a4162d9b4b8e98320acb64e33623" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa35dc091bd34f1a9085c6a7793fbe28" is_pre_election: true
I20260810 07:29:11.927354 9598 leader_election.cc:288] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28 [CANDIDATE]: Term 1 election: Requested vote from peers 8355a4162d9b4b8e98320acb64e33623 (127.12.157.194:38969), 76d191d270f7467dbdfbfa54b946a937 (127.12.157.193:39897)
I20260810 07:29:11.927491 9475 raft_consensus.cc:2379] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8355a4162d9b4b8e98320acb64e33623 in current term 1: Already voted for candidate fa35dc091bd34f1a9085c6a7793fbe28 in this term.
I20260810 07:29:11.927740 9256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3e2d733024479c920114cf844b3d2d" candidate_uuid: "fa35dc091bd34f1a9085c6a7793fbe28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8355a4162d9b4b8e98320acb64e33623"
I20260810 07:29:11.927786 9054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3e2d733024479c920114cf844b3d2d" candidate_uuid: "fa35dc091bd34f1a9085c6a7793fbe28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d191d270f7467dbdfbfa54b946a937"
I20260810 07:29:11.927872 9054 raft_consensus.cc:3037] T 6c3e2d733024479c920114cf844b3d2d P 76d191d270f7467dbdfbfa54b946a937 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:11.928444 9597 raft_consensus.cc:504] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8355a4162d9b4b8e98320acb64e33623" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38969 } } peers { permanent_uuid: "76d191d270f7467dbdfbfa54b946a937" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39897 } } peers { permanent_uuid: "fa35dc091bd34f1a9085c6a7793fbe28" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34353 } }
I20260810 07:29:11.929096 9054 raft_consensus.cc:2454] T 6c3e2d733024479c920114cf844b3d2d P 76d191d270f7467dbdfbfa54b946a937 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa35dc091bd34f1a9085c6a7793fbe28 in term 1.
I20260810 07:29:11.929361 9475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3e2d733024479c920114cf844b3d2d" candidate_uuid: "8355a4162d9b4b8e98320acb64e33623" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa35dc091bd34f1a9085c6a7793fbe28"
I20260810 07:29:11.929374 9052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3e2d733024479c920114cf844b3d2d" candidate_uuid: "8355a4162d9b4b8e98320acb64e33623" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d191d270f7467dbdfbfa54b946a937"
I20260810 07:29:11.929415 9369 leader_election.cc:302] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d191d270f7467dbdfbfa54b946a937, fa35dc091bd34f1a9085c6a7793fbe28; no voters:
I20260810 07:29:11.929526 9475 raft_consensus.cc:2379] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8355a4162d9b4b8e98320acb64e33623 in current term 1: Already voted for candidate fa35dc091bd34f1a9085c6a7793fbe28 in this term.
I20260810 07:29:11.928776 9256 raft_consensus.cc:2379] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fa35dc091bd34f1a9085c6a7793fbe28 in current term 1: Already voted for candidate 8355a4162d9b4b8e98320acb64e33623 in this term.
I20260810 07:29:11.929597 9052 raft_consensus.cc:2379] T 6c3e2d733024479c920114cf844b3d2d P 76d191d270f7467dbdfbfa54b946a937 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8355a4162d9b4b8e98320acb64e33623 in current term 1: Already voted for candidate fa35dc091bd34f1a9085c6a7793fbe28 in this term.
I20260810 07:29:11.929612 9598 raft_consensus.cc:2781] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:11.929824 9598 raft_consensus.cc:686] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28 [term 1 LEADER]: Becoming Leader. State: Replica: fa35dc091bd34f1a9085c6a7793fbe28, State: Running, Role: LEADER
I20260810 07:29:11.928850 9597 leader_election.cc:288] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623 [CANDIDATE]: Term 1 election: Requested vote from peers 76d191d270f7467dbdfbfa54b946a937 (127.12.157.193:39897), fa35dc091bd34f1a9085c6a7793fbe28 (127.12.157.195:34353)
I20260810 07:29:11.930122 9156 leader_election.cc:302] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8355a4162d9b4b8e98320acb64e33623; no voters: 76d191d270f7467dbdfbfa54b946a937, fa35dc091bd34f1a9085c6a7793fbe28
I20260810 07:29:11.930168 9598 consensus_queue.cc:237] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8355a4162d9b4b8e98320acb64e33623" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38969 } } peers { permanent_uuid: "76d191d270f7467dbdfbfa54b946a937" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39897 } } peers { permanent_uuid: "fa35dc091bd34f1a9085c6a7793fbe28" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34353 } }
I20260810 07:29:11.930275 9597 raft_consensus.cc:2726] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 07:29:11.933629 8811 catalog_manager.cc:5495] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa35dc091bd34f1a9085c6a7793fbe28 (127.12.157.195). New cstate: current_term: 1 leader_uuid: "fa35dc091bd34f1a9085c6a7793fbe28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8355a4162d9b4b8e98320acb64e33623" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76d191d270f7467dbdfbfa54b946a937" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa35dc091bd34f1a9085c6a7793fbe28" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34353 } health_report { overall_health: HEALTHY } } }
W20260810 07:29:11.937700 9559 tablet.cc:2369] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 07:29:11.942106 9339 tablet.cc:2369] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 07:29:11.972002 9124 tablet.cc:2369] T 6c3e2d733024479c920114cf844b3d2d P 76d191d270f7467dbdfbfa54b946a937: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:29:11.975721 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:11.971452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36066 (local address 127.12.157.195:34353)
0810 07:29:11.971614 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:11.971619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:11.971635 (+ 16us) server_negotiation.cc:407] Connection header received
0810 07:29:11.971723 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:11.971726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:11.971791 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:11.971910 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:29:11.972228 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.972902 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.974167 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:11.974398 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:11.974484 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:11.974843 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:29:11.975217 (+ 374us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:29:11.975306 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:11.975459 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 07:29:11.975549 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 07:29:11.981940 9256 raft_consensus.cc:1261] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623 [term 1 FOLLOWER]: Refusing update from remote peer fa35dc091bd34f1a9085c6a7793fbe28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:29:11.981952 9052 raft_consensus.cc:1261] T 6c3e2d733024479c920114cf844b3d2d P 76d191d270f7467dbdfbfa54b946a937 [term 1 FOLLOWER]: Refusing update from remote peer fa35dc091bd34f1a9085c6a7793fbe28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:29:11.982560 9605 consensus_queue.cc:1035] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8355a4162d9b4b8e98320acb64e33623" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 38969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:29:11.982758 9598 consensus_queue.cc:1035] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d191d270f7467dbdfbfa54b946a937" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:29:11.991282 9613 mvcc.cc:205] Tried to move back new op lower bound from 7316877115310800896 to 7316877115108831232. Current Snapshot: MvccSnapshot[applied={T|T < 7316877115310800896}]
I20260810 07:29:11.991509 9612 mvcc.cc:205] Tried to move back new op lower bound from 7316877115310800896 to 7316877115108831232. Current Snapshot: MvccSnapshot[applied={T|T < 7316877115310800896}]
I20260810 07:29:11.998324 8811 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:29:11.998505 8811 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:29:12.001134 8811 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c4f4121c7e6e41f9a9989f554bd03f35: Sending DeleteTablet for 3 replicas of tablet 6c3e2d733024479c920114cf844b3d2d
I20260810 07:29:12.002101 9019 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c3e2d733024479c920114cf844b3d2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:29:11 UTC) from {username='slave'} at 127.0.0.1:58128
I20260810 07:29:12.002154 9446 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c3e2d733024479c920114cf844b3d2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:29:11 UTC) from {username='slave'} at 127.0.0.1:36050
I20260810 07:29:12.002300 9226 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c3e2d733024479c920114cf844b3d2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:29:11 UTC) from {username='slave'} at 127.0.0.1:37666
I20260810 07:29:12.002791 9630 tablet_replica.cc:330] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623: stopping tablet replica
I20260810 07:29:12.002978 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 8907
I20260810 07:29:12.002821 9625 tablet_replica.cc:330] T 6c3e2d733024479c920114cf844b3d2d P 76d191d270f7467dbdfbfa54b946a937: stopping tablet replica
I20260810 07:29:12.003162 9630 raft_consensus.cc:2229] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:29:12.003542 9630 raft_consensus.cc:2258] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:29:12.011634 9629 tablet_replica.cc:330] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28: stopping tablet replica
I20260810 07:29:12.012151 9629 raft_consensus.cc:2229] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:29:12.012800 9629 raft_consensus.cc:2258] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:29:12.013213 9630 ts_tablet_manager.cc:1905] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:29:12.016630 9630 ts_tablet_manager.cc:1918] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:29:12.016763 9630 log.cc:1196] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-1/wal/wals/6c3e2d733024479c920114cf844b3d2d
I20260810 07:29:12.017148 9630 ts_tablet_manager.cc:1939] T 6c3e2d733024479c920114cf844b3d2d P 8355a4162d9b4b8e98320acb64e33623: Deleting consensus metadata
I20260810 07:29:12.017743 9629 ts_tablet_manager.cc:1905] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:29:12.018638 8790 catalog_manager.cc:4841] TS 8355a4162d9b4b8e98320acb64e33623 (127.12.157.194:38969): tablet 6c3e2d733024479c920114cf844b3d2d (table test-table [id=cbbc773376cc4baab716918f23fc962f]) successfully deleted
I20260810 07:29:12.022038 9629 ts_tablet_manager.cc:1918] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:29:12.022177 9629 log.cc:1196] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786346694975122-12919-0/minicluster-data/ts-2/wal/wals/6c3e2d733024479c920114cf844b3d2d
I20260810 07:29:12.022791 9629 ts_tablet_manager.cc:1939] T 6c3e2d733024479c920114cf844b3d2d P fa35dc091bd34f1a9085c6a7793fbe28: Deleting consensus metadata
I20260810 07:29:12.024497 8790 catalog_manager.cc:4841] TS fa35dc091bd34f1a9085c6a7793fbe28 (127.12.157.195:34353): tablet 6c3e2d733024479c920114cf844b3d2d (table test-table [id=cbbc773376cc4baab716918f23fc962f]) successfully deleted
W20260810 07:29:12.024636 8791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.157.193:39897 (error 108)
I20260810 07:29:12.025702 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 9131
I20260810 07:29:12.025959 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:12.025583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:39897 (local address 127.0.0.1:58140)
0810 07:29:12.025728 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:12.025842 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.193:39897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 07:29:12.037954 8791 catalog_manager.cc:4584] TS 76d191d270f7467dbdfbfa54b946a937 (127.12.157.193:39897): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6c3e2d733024479c920114cf844b3d2d: Network error: Client connection negotiation failed: client connection to 127.12.157.193:39897: connect: Connection refused (error 111)
I20260810 07:29:12.044871 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 9341
I20260810 07:29:12.057220 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 8765
2026-08-10T07:29:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1311 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:29:12 dist-test-slave-ffrd krb5kdc[9636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:29:12 dist-test-slave-ffrd krb5kdc[9636](info): set up 2 sockets
Aug 10 07:29:12 dist-test-slave-ffrd krb5kdc[9636](info): commencing operation
krb5kdc: starting...
W20260810 07:29:13.332787 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.242s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:29:13 dist-test-slave-ffrd krb5kdc[9636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:29:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:29:13.473400 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:41891
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:45943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:41891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:13.592806 9659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:13.593134 9659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:13.593210 9659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:13.597196 9659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:13.597299 9659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:13.597338 9659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:13.597378 9659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:13.597414 9659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:13.602813 9659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:45943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:41891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:41891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:13.603976 9659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:13.605063 9659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:13.613847 9668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:13.613842 9673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:13.613860 9670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:13.614390 9659 server_base.cc:1034] running on GCE node
I20260810 07:29:13.614996 9659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:13.616117 9659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:13.617332 9659 hybrid_clock.cc:648] HybridClock initialized: now 1786346953617303 us; error 40 us; skew 500 ppm
Aug 10 07:29:13 dist-test-slave-ffrd krb5kdc[9636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346953, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:13.624486 9659 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:29:13.625509 9659 webserver.cc:460] Webserver started at http://127.12.157.254:40299/ using document root <none> and password file <none>
I20260810 07:29:13.626076 9659 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:13.626142 9659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:13.626328 9659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:13.628401 9659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "cc99250996d0483bb935bec39a191e53"
format_stamp: "Formatted at 2026-08-10 07:29:13 on dist-test-slave-ffrd"
server_key: "548a4420cf4e214bc87e6a31804b027d"
server_key_iv: "efe9399f98a15eb4cf3533db6dd8bc8a"
server_key_version: "encryptionkey@0"
I20260810 07:29:13.628995 9659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "cc99250996d0483bb935bec39a191e53"
format_stamp: "Formatted at 2026-08-10 07:29:13 on dist-test-slave-ffrd"
server_key: "548a4420cf4e214bc87e6a31804b027d"
server_key_iv: "efe9399f98a15eb4cf3533db6dd8bc8a"
server_key_version: "encryptionkey@0"
I20260810 07:29:13.632786 9659 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 07:29:13.635291 9685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:13.636543 9659 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 07:29:13.636646 9659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "cc99250996d0483bb935bec39a191e53"
format_stamp: "Formatted at 2026-08-10 07:29:13 on dist-test-slave-ffrd"
server_key: "548a4420cf4e214bc87e6a31804b027d"
server_key_iv: "efe9399f98a15eb4cf3533db6dd8bc8a"
server_key_version: "encryptionkey@0"
I20260810 07:29:13.636732 9659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:13.666832 9659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:13.669951 9659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:13.670116 9659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:13.677851 9659 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:41891
I20260810 07:29:13.677871 9774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:41891 every 8 connection(s)
I20260810 07:29:13.678826 9659 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:13.681366 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 9659
I20260810 07:29:13.681520 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:29:13.681756 9780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:13.681797 12919 external_mini_cluster.cc:1428] Setting key 7ea06e0ae5640b61e254401baa612857
I20260810 07:29:13.688427 9780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc99250996d0483bb935bec39a191e53: Bootstrap starting.
Aug 10 07:29:13 dist-test-slave-ffrd krb5kdc[9636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:29:13.691090 9780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc99250996d0483bb935bec39a191e53: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:13.692037 9780 log.cc:824] T 00000000000000000000000000000000 P cc99250996d0483bb935bec39a191e53: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:13.694492 9780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc99250996d0483bb935bec39a191e53: No bootstrap required, opened a new log
I20260810 07:29:13.696530 9784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:13.683214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55046 (local address 127.12.157.254:41891)
0810 07:29:13.683698 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:13.683713 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:13.683748 (+ 35us) server_negotiation.cc:407] Connection header received
0810 07:29:13.684467 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:13.684485 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:13.684807 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:13.685197 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:13.686034 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:13.687067 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:13.687956 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:13.688433 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:13.688607 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:13.691333 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:13.691341 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:13.691364 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:13.691400 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:13.693399 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:13.693853 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:13.693860 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:13.693866 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:13.694000 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:13.694267 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:13.694271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:13.694273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:13.694787 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:13.694980 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:13.695272 (+ 292us) server_negotiation.cc:299] Negotiation successful
0810 07:29:13.695581 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":347,"thread_start_us":167,"threads_started":1}
I20260810 07:29:13.699039 9780 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc99250996d0483bb935bec39a191e53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc99250996d0483bb935bec39a191e53" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41891 } }
I20260810 07:29:13.699268 9780 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc99250996d0483bb935bec39a191e53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:13.699306 9780 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc99250996d0483bb935bec39a191e53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc99250996d0483bb935bec39a191e53, State: Initialized, Role: FOLLOWER
I20260810 07:29:13.699721 9780 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc99250996d0483bb935bec39a191e53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc99250996d0483bb935bec39a191e53" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41891 } }
I20260810 07:29:13.699853 9780 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc99250996d0483bb935bec39a191e53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:29:13.699895 9780 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc99250996d0483bb935bec39a191e53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:29:13.699962 9780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc99250996d0483bb935bec39a191e53 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:13.700860 9780 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc99250996d0483bb935bec39a191e53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc99250996d0483bb935bec39a191e53" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41891 } }
I20260810 07:29:13.701273 9780 leader_election.cc:302] T 00000000000000000000000000000000 P cc99250996d0483bb935bec39a191e53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc99250996d0483bb935bec39a191e53; no voters:
I20260810 07:29:13.701567 9780 leader_election.cc:288] T 00000000000000000000000000000000 P cc99250996d0483bb935bec39a191e53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:29:13.701714 9787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc99250996d0483bb935bec39a191e53 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:13.701977 9787 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc99250996d0483bb935bec39a191e53 [term 1 LEADER]: Becoming Leader. State: Replica: cc99250996d0483bb935bec39a191e53, State: Running, Role: LEADER
I20260810 07:29:13.702292 9787 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc99250996d0483bb935bec39a191e53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc99250996d0483bb935bec39a191e53" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41891 } }
I20260810 07:29:13.702953 9780 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc99250996d0483bb935bec39a191e53 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:13.704145 9789 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc99250996d0483bb935bec39a191e53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc99250996d0483bb935bec39a191e53. Latest consensus state: current_term: 1 leader_uuid: "cc99250996d0483bb935bec39a191e53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc99250996d0483bb935bec39a191e53" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41891 } } }
I20260810 07:29:13.704286 9789 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc99250996d0483bb935bec39a191e53 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:13.704504 9788 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc99250996d0483bb935bec39a191e53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc99250996d0483bb935bec39a191e53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc99250996d0483bb935bec39a191e53" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41891 } } }
I20260810 07:29:13.704609 9788 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc99250996d0483bb935bec39a191e53 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:13.705312 9799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:13.708467 9799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:13.714264 9799 catalog_manager.cc:1347] Generated new cluster ID: 9e0cb38876384ac7922b24be737268e8
I20260810 07:29:13.714336 9799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:13.734171 9799 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:13.735368 9799 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:13.742359 9799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc99250996d0483bb935bec39a191e53: Generated new TSK 0
I20260810 07:29:13.743108 9799 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
I20260810 07:29:13.788795 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:0
--local_ip_for_outbound_sockets=127.12.157.193
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:41891
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:45943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:13.910559 9822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:13.910857 9822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:13.910943 9822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:13.914942 9822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:13.915021 9822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:13.915112 9822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:29:13.920287 9822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:45943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:41891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:13.921419 9822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:13.922427 9822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 07:29:13.932161 9822 server_base.cc:1034] running on GCE node
W20260810 07:29:13.932106 9836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:13.932090 9834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:13.932117 9832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:13.932720 9822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:13.933416 9822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:13.934613 9822 hybrid_clock.cc:648] HybridClock initialized: now 1786346953934599 us; error 26 us; skew 500 ppm
Aug 10 07:29:13 dist-test-slave-ffrd krb5kdc[9636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346953, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:13.937610 9822 init.cc:377] Logged in from keytab as kudu/127.12.157.193@KRBTEST.COM (short username kudu)
I20260810 07:29:13.938511 9822 webserver.cc:460] Webserver started at http://127.12.157.193:45623/ using document root <none> and password file <none>
I20260810 07:29:13.939090 9822 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:13.939183 9822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:13.939425 9822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:13.941748 9822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-0/data/instance:
uuid: "cff7321d643740eab7241f3549bea21b"
format_stamp: "Formatted at 2026-08-10 07:29:13 on dist-test-slave-ffrd"
server_key: "9088b626014ce738c096316031f556c6"
server_key_iv: "16c7ad4a493554381bcf465c29096d07"
server_key_version: "encryptionkey@0"
I20260810 07:29:13.942349 9822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance:
uuid: "cff7321d643740eab7241f3549bea21b"
format_stamp: "Formatted at 2026-08-10 07:29:13 on dist-test-slave-ffrd"
server_key: "9088b626014ce738c096316031f556c6"
server_key_iv: "16c7ad4a493554381bcf465c29096d07"
server_key_version: "encryptionkey@0"
I20260810 07:29:13.946197 9822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 07:29:13.948792 9847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:13.950165 9822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:29:13.950341 9822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "cff7321d643740eab7241f3549bea21b"
format_stamp: "Formatted at 2026-08-10 07:29:13 on dist-test-slave-ffrd"
server_key: "9088b626014ce738c096316031f556c6"
server_key_iv: "16c7ad4a493554381bcf465c29096d07"
server_key_version: "encryptionkey@0"
I20260810 07:29:13.950471 9822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:13.969955 9822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:13.973356 9822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:13.973568 9822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:13.974200 9822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:29:13.975064 9822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:29:13.975138 9822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:13.975210 9822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:29:13.975255 9822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:13.984912 9822 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:44885
I20260810 07:29:13.985006 10040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:44885 every 8 connection(s)
I20260810 07:29:13.985901 9822 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:29:13.987411 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 9822
I20260810 07:29:13.987545 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance
I20260810 07:29:13.987886 12919 external_mini_cluster.cc:1428] Setting key baa29c0c2b66cd12eabc1b4a1bdf7cec
Aug 10 07:29:13 dist-test-slave-ffrd krb5kdc[9636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346953, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.157.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.194@KRBTEST.COM" created.
I20260810 07:29:14.002055 9784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:13.987886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:45495 (local address 127.12.157.254:41891)
0810 07:29:13.988047 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:13.988052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:13.989169 (+ 1117us) server_negotiation.cc:407] Connection header received
0810 07:29:13.990722 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:13.990726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:13.990784 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:13.990881 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:13.992487 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:13.993187 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:13.994459 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:13.994720 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:13.994844 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:13.997792 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:13.997796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:13.997799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:13.997824 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:13.999541 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:14.000228 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:14.000232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:14.000234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:14.000324 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:14.000699 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:14.000703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:14.000705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:14.000951 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:14.001111 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:14.001718 (+ 607us) server_negotiation.cc:299] Negotiation successful
0810 07:29:14.001878 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 07:29:14.003029 10044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:13.988392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:41891 (local address 127.12.157.193:45495)
0810 07:29:13.989005 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:13.989063 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:13.990489 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:13.991071 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:13.991078 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:13.991496 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:13.992261 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:13.992284 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:13.993325 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:13.993328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:13.994308 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:13.994318 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:13.994896 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:13.994900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:13.995087 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:13.995746 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:13.995764 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:13.997552 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:13.999730 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:13.999737 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:13.999754 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:14.000111 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:14.000433 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:14.000436 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:14.000438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:14.000594 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:14.001132 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:14.001140 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:14.001421 (+ 281us) client_negotiation.cc:769] Sending connection context
0810 07:29:14.001673 (+ 252us) client_negotiation.cc:240] Negotiation successful
0810 07:29:14.002058 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":160,"threads_started":1}
I20260810 07:29:14.004616 10041 heartbeater.cc:347] Connected to a master server at 127.12.157.254:41891
I20260810 07:29:14.004920 10041 heartbeater.cc:464] Registering TS with master...
I20260810 07:29:14.005631 10041 heartbeater.cc:511] Master 127.12.157.254:41891 requested a full tablet report, sending...
I20260810 07:29:14.007323 9713 ts_manager.cc:194] Registered new tserver with Master: cff7321d643740eab7241f3549bea21b (127.12.157.193:44885)
I20260810 07:29:14.008657 9713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.193@KRBTEST.COM'} at 127.12.157.193:45495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
I20260810 07:29:14.020347 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:0
--local_ip_for_outbound_sockets=127.12.157.194
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:41891
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:45943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:14.136493 10050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:14.136761 10050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:14.136806 10050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:14.141202 10050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:14.141275 10050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:14.141337 10050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:29:14.147138 10050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:45943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:41891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:14.148355 10050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:14.149425 10050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:14.159605 10063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:14.159614 10056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:14.159623 10055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:14.159902 10050 server_base.cc:1034] running on GCE node
I20260810 07:29:14.160326 10050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:14.161095 10050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:14.162258 10050 hybrid_clock.cc:648] HybridClock initialized: now 1786346954162240 us; error 31 us; skew 500 ppm
Aug 10 07:29:14 dist-test-slave-ffrd krb5kdc[9636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346954, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:14.165171 10050 init.cc:377] Logged in from keytab as kudu/127.12.157.194@KRBTEST.COM (short username kudu)
I20260810 07:29:14.166049 10050 webserver.cc:460] Webserver started at http://127.12.157.194:45899/ using document root <none> and password file <none>
I20260810 07:29:14.166572 10050 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:14.166630 10050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:14.166791 10050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:14.168787 10050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-1/data/instance:
uuid: "efe0b52cd1eb4f80a60a6baeeb5acc4a"
format_stamp: "Formatted at 2026-08-10 07:29:14 on dist-test-slave-ffrd"
server_key: "46f3e6f0fda4189f69fe4f1b7aec13d1"
server_key_iv: "b047781e2621c04b34221116116e85d7"
server_key_version: "encryptionkey@0"
I20260810 07:29:14.169417 10050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance:
uuid: "efe0b52cd1eb4f80a60a6baeeb5acc4a"
format_stamp: "Formatted at 2026-08-10 07:29:14 on dist-test-slave-ffrd"
server_key: "46f3e6f0fda4189f69fe4f1b7aec13d1"
server_key_iv: "b047781e2621c04b34221116116e85d7"
server_key_version: "encryptionkey@0"
I20260810 07:29:14.173228 10050 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 07:29:14.175683 10080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:14.176872 10050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:29:14.177053 10050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "efe0b52cd1eb4f80a60a6baeeb5acc4a"
format_stamp: "Formatted at 2026-08-10 07:29:14 on dist-test-slave-ffrd"
server_key: "46f3e6f0fda4189f69fe4f1b7aec13d1"
server_key_iv: "b047781e2621c04b34221116116e85d7"
server_key_version: "encryptionkey@0"
I20260810 07:29:14.177172 10050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:14.189764 10050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:14.193419 10050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:14.193625 10050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:14.194247 10050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:29:14.195149 10050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:29:14.195222 10050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:14.195295 10050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:29:14.195345 10050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:14.205629 10050 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:37301
I20260810 07:29:14.205739 10255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:37301 every 8 connection(s)
I20260810 07:29:14.206632 10050 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:29:14.208719 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 10050
I20260810 07:29:14.209249 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance
I20260810 07:29:14.209615 12919 external_mini_cluster.cc:1428] Setting key 6cd9ccdad78e32b543d4653150c639fb
Aug 10 07:29:14 dist-test-slave-ffrd krb5kdc[9636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346954, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.157.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.195@KRBTEST.COM" created.
I20260810 07:29:14.224388 9784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:14.209703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:57803 (local address 127.12.157.254:41891)
0810 07:29:14.209917 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:14.209921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:14.210984 (+ 1063us) server_negotiation.cc:407] Connection header received
0810 07:29:14.212866 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:14.212870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:14.212933 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:14.213042 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:14.214317 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.214869 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.216701 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.217029 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.217125 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:14.220245 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:14.220249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:14.220251 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:14.220277 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:14.221971 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:14.222545 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:14.222548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:14.222549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:14.222638 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:14.222997 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:14.222999 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:14.223001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:14.223199 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:14.223278 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:14.223913 (+ 635us) server_negotiation.cc:299] Negotiation successful
0810 07:29:14.224092 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 07:29:14.225230 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:14.210041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:41891 (local address 127.12.157.194:57803)
0810 07:29:14.210681 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:14.210848 (+ 167us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:14.212560 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:14.213234 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:14.213242 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:14.213621 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:14.214151 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:14.214164 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.215012 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.215021 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:14.216550 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:14.216566 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.217210 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.217216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:14.217471 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:14.218158 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:14.218186 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:14.220055 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:14.222103 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:14.222110 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:14.222126 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:14.222395 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:14.222743 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:14.222746 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:14.222747 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:14.222899 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:14.223304 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:14.223310 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:14.223632 (+ 322us) client_negotiation.cc:769] Sending connection context
0810 07:29:14.223800 (+ 168us) client_negotiation.cc:240] Negotiation successful
0810 07:29:14.224163 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":133,"threads_started":1}
I20260810 07:29:14.226550 10256 heartbeater.cc:347] Connected to a master server at 127.12.157.254:41891
I20260810 07:29:14.226801 10256 heartbeater.cc:464] Registering TS with master...
I20260810 07:29:14.227324 10256 heartbeater.cc:511] Master 127.12.157.254:41891 requested a full tablet report, sending...
I20260810 07:29:14.228478 9713 ts_manager.cc:194] Registered new tserver with Master: efe0b52cd1eb4f80a60a6baeeb5acc4a (127.12.157.194:37301)
I20260810 07:29:14.229125 9713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.194@KRBTEST.COM'} at 127.12.157.194:57803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
I20260810 07:29:14.242239 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:0
--local_ip_for_outbound_sockets=127.12.157.195
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:41891
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:45943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:14.361907 10263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:14.362205 10263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:14.362291 10263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:14.366365 10263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:14.366458 10263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:14.366544 10263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:29:14.371950 10263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:45943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:41891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:14.373176 10263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:14.374388 10263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:14.384483 10274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:14.384449 10270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:14.384472 10272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:14.384696 10263 server_base.cc:1034] running on GCE node
I20260810 07:29:14.385188 10263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:14.385855 10263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:14.387048 10263 hybrid_clock.cc:648] HybridClock initialized: now 1786346954387036 us; error 24 us; skew 500 ppm
Aug 10 07:29:14 dist-test-slave-ffrd krb5kdc[9636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346954, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:14.390102 10263 init.cc:377] Logged in from keytab as kudu/127.12.157.195@KRBTEST.COM (short username kudu)
I20260810 07:29:14.391100 10263 webserver.cc:460] Webserver started at http://127.12.157.195:38613/ using document root <none> and password file <none>
I20260810 07:29:14.391764 10263 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:14.391839 10263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:14.392051 10263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:14.394164 10263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-2/data/instance:
uuid: "eb40556e6b444b44a4c61581ce6a4b06"
format_stamp: "Formatted at 2026-08-10 07:29:14 on dist-test-slave-ffrd"
server_key: "5e182e23df0e0066602da816aa0db02a"
server_key_iv: "ccc811eff3a65d007d837f266a2e49ea"
server_key_version: "encryptionkey@0"
I20260810 07:29:14.394682 10263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance:
uuid: "eb40556e6b444b44a4c61581ce6a4b06"
format_stamp: "Formatted at 2026-08-10 07:29:14 on dist-test-slave-ffrd"
server_key: "5e182e23df0e0066602da816aa0db02a"
server_key_iv: "ccc811eff3a65d007d837f266a2e49ea"
server_key_version: "encryptionkey@0"
I20260810 07:29:14.398468 10263 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 07:29:14.401017 10284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:14.402257 10263 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:29:14.402359 10263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "eb40556e6b444b44a4c61581ce6a4b06"
format_stamp: "Formatted at 2026-08-10 07:29:14 on dist-test-slave-ffrd"
server_key: "5e182e23df0e0066602da816aa0db02a"
server_key_iv: "ccc811eff3a65d007d837f266a2e49ea"
server_key_version: "encryptionkey@0"
I20260810 07:29:14.402452 10263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:14.415581 10263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:14.418962 10263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:14.419121 10263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:14.419745 10263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:29:14.420540 10263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:29:14.420588 10263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:14.420627 10263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:29:14.420642 10263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:14.429831 10263 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:33539
I20260810 07:29:14.429930 10458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:33539 every 8 connection(s)
I20260810 07:29:14.430758 10263 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:29:14.440006 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 10263
I20260810 07:29:14.440156 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance
I20260810 07:29:14.440516 12919 external_mini_cluster.cc:1428] Setting key 74320409f5242a4c4a07823c80279a00
Aug 10 07:29:14 dist-test-slave-ffrd krb5kdc[9636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346954, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:29:14.445703 9784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:14.432771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:57311 (local address 127.12.157.254:41891)
0810 07:29:14.433018 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:14.433023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:14.433841 (+ 818us) server_negotiation.cc:407] Connection header received
0810 07:29:14.435241 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:14.435245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:14.435300 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:14.435415 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:14.436612 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.437259 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.438432 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.438658 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.438797 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:14.442067 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:14.442094 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:14.442097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:14.442125 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:14.443722 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:14.444239 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:14.444242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:14.444243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:14.444319 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:14.444673 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:14.444676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:14.444677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:14.444845 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:14.444924 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:14.445487 (+ 563us) server_negotiation.cc:299] Negotiation successful
0810 07:29:14.445586 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 07:29:14.446549 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:14.433194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:41891 (local address 127.12.157.195:57311)
0810 07:29:14.433676 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:14.433721 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:14.435024 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:14.435591 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:14.435598 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:14.435968 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:14.436455 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:14.436470 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.437397 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.437400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:14.438295 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:14.438305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.438855 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.438860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:14.439036 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:14.439659 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:14.439677 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:14.441877 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:14.443857 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:14.443866 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:14.443878 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:14.444146 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:14.444430 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:14.444433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:14.444435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:14.444581 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:14.445006 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:14.445013 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:14.445272 (+ 259us) client_negotiation.cc:769] Sending connection context
0810 07:29:14.445434 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 07:29:14.445686 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":115,"threads_started":1}
I20260810 07:29:14.447762 10459 heartbeater.cc:347] Connected to a master server at 127.12.157.254:41891
I20260810 07:29:14.448038 10459 heartbeater.cc:464] Registering TS with master...
I20260810 07:29:14.448550 10459 heartbeater.cc:511] Master 127.12.157.254:41891 requested a full tablet report, sending...
I20260810 07:29:14.449645 9713 ts_manager.cc:194] Registered new tserver with Master: eb40556e6b444b44a4c61581ce6a4b06 (127.12.157.195:33539)
I20260810 07:29:14.450263 9713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.195@KRBTEST.COM'} at 127.12.157.195:57311
I20260810 07:29:14.454838 12919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:29:14.464991 9784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:14.456930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55048 (local address 127.12.157.254:41891)
0810 07:29:14.457102 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:14.457106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:14.457354 (+ 248us) server_negotiation.cc:407] Connection header received
0810 07:29:14.457450 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:14.457452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:14.457495 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:14.457574 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:14.458292 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.458844 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.459695 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.459915 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.459993 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:14.461633 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:14.461637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:14.461639 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:14.461661 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:14.463143 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:14.463687 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:14.463690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:14.463691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:14.463773 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:14.464124 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:14.464126 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:14.464127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:14.464337 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:14.464471 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:14.464742 (+ 271us) server_negotiation.cc:299] Negotiation successful
0810 07:29:14.464845 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260810 07:29:14.468245 9713 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:29:14.471503 9713 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:29:14.486996 10477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:14.480832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54358 (local address 127.12.157.193:44885)
0810 07:29:14.481670 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:14.481688 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:14.481714 (+ 26us) server_negotiation.cc:407] Connection header received
0810 07:29:14.481798 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:14.481804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:14.482037 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:14.482196 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:14.482647 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.483798 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.485633 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.486292 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.486439 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:14.486585 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:14.486732 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 07:29:14.486830 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":113,"threads_started":1}
I20260810 07:29:14.487412 10482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:14.481495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60796 (local address 127.12.157.194:37301)
0810 07:29:14.482368 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:14.482394 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:14.482419 (+ 25us) server_negotiation.cc:407] Connection header received
0810 07:29:14.482496 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:14.482501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:14.482672 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:14.482864 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:14.483102 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.483950 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.485574 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.486466 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.486999 (+ 533us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:14.487128 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:14.487180 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 07:29:14.487272 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770,"thread_start_us":152,"threads_started":1}
I20260810 07:29:14.487540 10484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:14.481855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34218 (local address 127.12.157.195:33539)
0810 07:29:14.482471 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:14.482484 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:14.482503 (+ 19us) server_negotiation.cc:407] Connection header received
0810 07:29:14.482567 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:14.482571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:14.482716 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:14.482861 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:14.483329 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.484421 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.485830 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.486662 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.486773 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:14.486897 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:14.487279 (+ 382us) server_negotiation.cc:299] Negotiation successful
0810 07:29:14.487383 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":175,"threads_started":1}
I20260810 07:29:14.487476 10480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:14.481409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:33539 (local address 127.0.0.1:34218)
0810 07:29:14.481833 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:14.481845 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:14.481937 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:14.483056 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:14.483062 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:14.483083 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:14.483190 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:14.483196 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.484571 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.484574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:14.485709 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:14.485718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.486961 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.486964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:14.487052 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:14.487072 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 07:29:14.487142 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 07:29:14.487208 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":22,"thread_start_us":85,"threads_started":1}
I20260810 07:29:14.488679 10478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:14.481018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:37301 (local address 127.0.0.1:60796)
0810 07:29:14.481546 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:14.481560 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:14.481651 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:14.482850 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:14.482855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:14.482878 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:14.482975 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:14.482981 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.484090 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.484094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:14.485430 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:14.485444 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.486616 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.486622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:14.486732 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:14.486759 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 07:29:14.486807 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 07:29:14.486892 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":241,"thread_start_us":114,"threads_started":1}
I20260810 07:29:14.488971 10476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:14.480687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:44885 (local address 127.0.0.1:54358)
0810 07:29:14.481039 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:14.481086 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:14.481215 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:14.482272 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:14.482278 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:14.482301 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:14.482504 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:14.482514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.483966 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.483972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:14.485482 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:14.485491 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.486426 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.486429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:14.486532 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:14.486569 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 07:29:14.486688 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 07:29:14.486812 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":40,"thread_start_us":74,"threads_started":1}
I20260810 07:29:14.490777 10364 tablet_service.cc:1467] Processing CreateTablet for tablet 095ae8d57798438c87c1243e6f866b38 (DEFAULT_TABLE table=test-table [id=b3a4063ebb8d4c2488c3d1e16bac3fe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:29:14.491762 10364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 095ae8d57798438c87c1243e6f866b38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:14.491541 10149 tablet_service.cc:1467] Processing CreateTablet for tablet 095ae8d57798438c87c1243e6f866b38 (DEFAULT_TABLE table=test-table [id=b3a4063ebb8d4c2488c3d1e16bac3fe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:29:14.491537 9929 tablet_service.cc:1467] Processing CreateTablet for tablet 095ae8d57798438c87c1243e6f866b38 (DEFAULT_TABLE table=test-table [id=b3a4063ebb8d4c2488c3d1e16bac3fe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:29:14.492525 9929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 095ae8d57798438c87c1243e6f866b38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:14.492525 10149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 095ae8d57798438c87c1243e6f866b38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:14.499250 10486 tablet_bootstrap.cc:492] T 095ae8d57798438c87c1243e6f866b38 P eb40556e6b444b44a4c61581ce6a4b06: Bootstrap starting.
I20260810 07:29:14.499467 10491 tablet_bootstrap.cc:492] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a: Bootstrap starting.
I20260810 07:29:14.500509 10492 tablet_bootstrap.cc:492] T 095ae8d57798438c87c1243e6f866b38 P cff7321d643740eab7241f3549bea21b: Bootstrap starting.
I20260810 07:29:14.501820 10486 tablet_bootstrap.cc:654] T 095ae8d57798438c87c1243e6f866b38 P eb40556e6b444b44a4c61581ce6a4b06: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:14.501821 10491 tablet_bootstrap.cc:654] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:14.502398 10492 tablet_bootstrap.cc:654] T 095ae8d57798438c87c1243e6f866b38 P cff7321d643740eab7241f3549bea21b: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:14.503412 10491 log.cc:824] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:14.503435 10486 log.cc:824] T 095ae8d57798438c87c1243e6f866b38 P eb40556e6b444b44a4c61581ce6a4b06: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:14.503576 10492 log.cc:824] T 095ae8d57798438c87c1243e6f866b38 P cff7321d643740eab7241f3549bea21b: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:14.505728 10492 tablet_bootstrap.cc:492] T 095ae8d57798438c87c1243e6f866b38 P cff7321d643740eab7241f3549bea21b: No bootstrap required, opened a new log
I20260810 07:29:14.505754 10491 tablet_bootstrap.cc:492] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a: No bootstrap required, opened a new log
I20260810 07:29:14.505754 10486 tablet_bootstrap.cc:492] T 095ae8d57798438c87c1243e6f866b38 P eb40556e6b444b44a4c61581ce6a4b06: No bootstrap required, opened a new log
I20260810 07:29:14.505959 10491 ts_tablet_manager.cc:1397] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 07:29:14.505957 10486 ts_tablet_manager.cc:1397] T 095ae8d57798438c87c1243e6f866b38 P eb40556e6b444b44a4c61581ce6a4b06: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 07:29:14.505998 10492 ts_tablet_manager.cc:1397] T 095ae8d57798438c87c1243e6f866b38 P cff7321d643740eab7241f3549bea21b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 07:29:14.509177 10492 raft_consensus.cc:348] T 095ae8d57798438c87c1243e6f866b38 P cff7321d643740eab7241f3549bea21b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7321d643740eab7241f3549bea21b" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 44885 } } peers { permanent_uuid: "eb40556e6b444b44a4c61581ce6a4b06" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33539 } } peers { permanent_uuid: "efe0b52cd1eb4f80a60a6baeeb5acc4a" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37301 } }
I20260810 07:29:14.509429 10492 raft_consensus.cc:374] T 095ae8d57798438c87c1243e6f866b38 P cff7321d643740eab7241f3549bea21b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:14.509495 10492 raft_consensus.cc:729] T 095ae8d57798438c87c1243e6f866b38 P cff7321d643740eab7241f3549bea21b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff7321d643740eab7241f3549bea21b, State: Initialized, Role: FOLLOWER
I20260810 07:29:14.509967 10492 consensus_queue.cc:260] T 095ae8d57798438c87c1243e6f866b38 P cff7321d643740eab7241f3549bea21b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7321d643740eab7241f3549bea21b" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 44885 } } peers { permanent_uuid: "eb40556e6b444b44a4c61581ce6a4b06" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33539 } } peers { permanent_uuid: "efe0b52cd1eb4f80a60a6baeeb5acc4a" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37301 } }
I20260810 07:29:14.509900 10491 raft_consensus.cc:348] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7321d643740eab7241f3549bea21b" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 44885 } } peers { permanent_uuid: "eb40556e6b444b44a4c61581ce6a4b06" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33539 } } peers { permanent_uuid: "efe0b52cd1eb4f80a60a6baeeb5acc4a" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37301 } }
I20260810 07:29:14.509895 10486 raft_consensus.cc:348] T 095ae8d57798438c87c1243e6f866b38 P eb40556e6b444b44a4c61581ce6a4b06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7321d643740eab7241f3549bea21b" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 44885 } } peers { permanent_uuid: "eb40556e6b444b44a4c61581ce6a4b06" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33539 } } peers { permanent_uuid: "efe0b52cd1eb4f80a60a6baeeb5acc4a" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37301 } }
I20260810 07:29:14.510188 10491 raft_consensus.cc:374] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:14.510188 10486 raft_consensus.cc:374] T 095ae8d57798438c87c1243e6f866b38 P eb40556e6b444b44a4c61581ce6a4b06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:14.510242 10491 raft_consensus.cc:729] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efe0b52cd1eb4f80a60a6baeeb5acc4a, State: Initialized, Role: FOLLOWER
I20260810 07:29:14.510253 10486 raft_consensus.cc:729] T 095ae8d57798438c87c1243e6f866b38 P eb40556e6b444b44a4c61581ce6a4b06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb40556e6b444b44a4c61581ce6a4b06, State: Initialized, Role: FOLLOWER
I20260810 07:29:14.510680 10491 consensus_queue.cc:260] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7321d643740eab7241f3549bea21b" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 44885 } } peers { permanent_uuid: "eb40556e6b444b44a4c61581ce6a4b06" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33539 } } peers { permanent_uuid: "efe0b52cd1eb4f80a60a6baeeb5acc4a" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37301 } }
I20260810 07:29:14.510680 10486 consensus_queue.cc:260] T 095ae8d57798438c87c1243e6f866b38 P eb40556e6b444b44a4c61581ce6a4b06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7321d643740eab7241f3549bea21b" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 44885 } } peers { permanent_uuid: "eb40556e6b444b44a4c61581ce6a4b06" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33539 } } peers { permanent_uuid: "efe0b52cd1eb4f80a60a6baeeb5acc4a" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37301 } }
I20260810 07:29:14.511051 10492 ts_tablet_manager.cc:1428] T 095ae8d57798438c87c1243e6f866b38 P cff7321d643740eab7241f3549bea21b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 07:29:14.511426 10041 heartbeater.cc:503] Master 127.12.157.254:41891 was elected leader, sending a full tablet report...
I20260810 07:29:14.511610 10486 ts_tablet_manager.cc:1428] T 095ae8d57798438c87c1243e6f866b38 P eb40556e6b444b44a4c61581ce6a4b06: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260810 07:29:14.511610 10491 ts_tablet_manager.cc:1428] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 07:29:14.511950 10459 heartbeater.cc:503] Master 127.12.157.254:41891 was elected leader, sending a full tablet report...
I20260810 07:29:14.512802 10256 heartbeater.cc:503] Master 127.12.157.254:41891 was elected leader, sending a full tablet report...
I20260810 07:29:14.578112 10503 raft_consensus.cc:482] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:29:14.578397 10503 raft_consensus.cc:504] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7321d643740eab7241f3549bea21b" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 44885 } } peers { permanent_uuid: "eb40556e6b444b44a4c61581ce6a4b06" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33539 } } peers { permanent_uuid: "efe0b52cd1eb4f80a60a6baeeb5acc4a" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37301 } }
I20260810 07:29:14.579739 10503 leader_election.cc:288] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cff7321d643740eab7241f3549bea21b (127.12.157.193:44885), eb40556e6b444b44a4c61581ce6a4b06 (127.12.157.195:33539)
I20260810 07:29:14.583863 10259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:14.579954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:44885 (local address 127.12.157.194:42435)
0810 07:29:14.580156 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:14.580174 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:14.580272 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:14.580695 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:14.580698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:14.580718 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:14.580852 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:14.580858 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.581804 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.581807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:14.582710 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:14.582720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.583421 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.583425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:14.583541 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:14.583564 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 07:29:14.583646 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 07:29:14.583703 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":21}
I20260810 07:29:14.583914 10477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:14.580128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:42435 (local address 127.12.157.193:44885)
0810 07:29:14.580430 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:14.580435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:14.580452 (+ 17us) server_negotiation.cc:407] Connection header received
0810 07:29:14.580508 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:14.580511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:14.580578 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:14.580680 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:14.581017 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.581696 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.582845 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.583296 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.583393 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:14.583433 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:14.583676 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 07:29:14.583761 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260810 07:29:14.583974 10484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:14.580431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:58207 (local address 127.12.157.195:33539)
0810 07:29:14.580555 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:14.580560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:14.580792 (+ 232us) server_negotiation.cc:407] Connection header received
0810 07:29:14.580856 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:14.580859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:14.580915 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:14.581013 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:14.581345 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.581901 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.582981 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.583417 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.583535 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:14.583585 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:14.583780 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 07:29:14.583850 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 07:29:14.584030 10505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:14.580266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:33539 (local address 127.12.157.194:58207)
0810 07:29:14.580665 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:14.580678 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:14.580779 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:14.581019 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:14.581022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:14.581040 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:14.581184 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:14.581191 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.582002 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.582005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:14.582851 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:14.582861 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.583535 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.583538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:14.583642 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:14.583668 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 07:29:14.583743 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 07:29:14.583801 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":16,"thread_start_us":136,"threads_started":1}
I20260810 07:29:14.584573 9965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "095ae8d57798438c87c1243e6f866b38" candidate_uuid: "efe0b52cd1eb4f80a60a6baeeb5acc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff7321d643740eab7241f3549bea21b" is_pre_election: true
I20260810 07:29:14.584601 10393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "095ae8d57798438c87c1243e6f866b38" candidate_uuid: "efe0b52cd1eb4f80a60a6baeeb5acc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb40556e6b444b44a4c61581ce6a4b06" is_pre_election: true
I20260810 07:29:14.584877 9965 raft_consensus.cc:2454] T 095ae8d57798438c87c1243e6f866b38 P cff7321d643740eab7241f3549bea21b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efe0b52cd1eb4f80a60a6baeeb5acc4a in term 0.
I20260810 07:29:14.584885 10393 raft_consensus.cc:2454] T 095ae8d57798438c87c1243e6f866b38 P eb40556e6b444b44a4c61581ce6a4b06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efe0b52cd1eb4f80a60a6baeeb5acc4a in term 0.
I20260810 07:29:14.585405 10082 leader_election.cc:302] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eb40556e6b444b44a4c61581ce6a4b06, efe0b52cd1eb4f80a60a6baeeb5acc4a; no voters:
I20260810 07:29:14.585634 10503 raft_consensus.cc:2781] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:29:14.585722 10503 raft_consensus.cc:482] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:29:14.585753 10503 raft_consensus.cc:3037] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:14.586850 10503 raft_consensus.cc:504] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7321d643740eab7241f3549bea21b" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 44885 } } peers { permanent_uuid: "eb40556e6b444b44a4c61581ce6a4b06" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33539 } } peers { permanent_uuid: "efe0b52cd1eb4f80a60a6baeeb5acc4a" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37301 } }
I20260810 07:29:14.587196 10503 leader_election.cc:288] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a [CANDIDATE]: Term 1 election: Requested vote from peers cff7321d643740eab7241f3549bea21b (127.12.157.193:44885), eb40556e6b444b44a4c61581ce6a4b06 (127.12.157.195:33539)
I20260810 07:29:14.587603 9965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "095ae8d57798438c87c1243e6f866b38" candidate_uuid: "efe0b52cd1eb4f80a60a6baeeb5acc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cff7321d643740eab7241f3549bea21b"
I20260810 07:29:14.587641 10393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "095ae8d57798438c87c1243e6f866b38" candidate_uuid: "efe0b52cd1eb4f80a60a6baeeb5acc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb40556e6b444b44a4c61581ce6a4b06"
I20260810 07:29:14.587707 9965 raft_consensus.cc:3037] T 095ae8d57798438c87c1243e6f866b38 P cff7321d643740eab7241f3549bea21b [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:14.587780 10393 raft_consensus.cc:3037] T 095ae8d57798438c87c1243e6f866b38 P eb40556e6b444b44a4c61581ce6a4b06 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:14.588801 9965 raft_consensus.cc:2454] T 095ae8d57798438c87c1243e6f866b38 P cff7321d643740eab7241f3549bea21b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efe0b52cd1eb4f80a60a6baeeb5acc4a in term 1.
I20260810 07:29:14.588932 10393 raft_consensus.cc:2454] T 095ae8d57798438c87c1243e6f866b38 P eb40556e6b444b44a4c61581ce6a4b06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efe0b52cd1eb4f80a60a6baeeb5acc4a in term 1.
I20260810 07:29:14.589169 10082 leader_election.cc:302] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cff7321d643740eab7241f3549bea21b, efe0b52cd1eb4f80a60a6baeeb5acc4a; no voters:
I20260810 07:29:14.589334 10503 raft_consensus.cc:2781] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:14.589588 10503 raft_consensus.cc:686] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a [term 1 LEADER]: Becoming Leader. State: Replica: efe0b52cd1eb4f80a60a6baeeb5acc4a, State: Running, Role: LEADER
I20260810 07:29:14.589922 10503 consensus_queue.cc:237] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7321d643740eab7241f3549bea21b" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 44885 } } peers { permanent_uuid: "eb40556e6b444b44a4c61581ce6a4b06" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33539 } } peers { permanent_uuid: "efe0b52cd1eb4f80a60a6baeeb5acc4a" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37301 } }
I20260810 07:29:14.593225 9709 catalog_manager.cc:5495] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a reported cstate change: term changed from 0 to 1, leader changed from <none> to efe0b52cd1eb4f80a60a6baeeb5acc4a (127.12.157.194). New cstate: current_term: 1 leader_uuid: "efe0b52cd1eb4f80a60a6baeeb5acc4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cff7321d643740eab7241f3549bea21b" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 44885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb40556e6b444b44a4c61581ce6a4b06" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efe0b52cd1eb4f80a60a6baeeb5acc4a" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 37301 } health_report { overall_health: HEALTHY } } }
I20260810 07:29:14.612911 10482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:14.609248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60800 (local address 127.12.157.194:37301)
0810 07:29:14.609388 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:14.609392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:14.609407 (+ 15us) server_negotiation.cc:407] Connection header received
0810 07:29:14.609530 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:14.609533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:14.609590 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:14.609680 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:29:14.609981 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.610491 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.611494 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.611706 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.611797 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:14.612059 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:29:14.612478 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:29:14.612590 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:14.612716 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 07:29:14.612766 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 07:29:14.619110 9965 raft_consensus.cc:1261] T 095ae8d57798438c87c1243e6f866b38 P cff7321d643740eab7241f3549bea21b [term 1 FOLLOWER]: Refusing update from remote peer efe0b52cd1eb4f80a60a6baeeb5acc4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:29:14.619110 10393 raft_consensus.cc:1261] T 095ae8d57798438c87c1243e6f866b38 P eb40556e6b444b44a4c61581ce6a4b06 [term 1 FOLLOWER]: Refusing update from remote peer efe0b52cd1eb4f80a60a6baeeb5acc4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:29:14.619895 10506 consensus_queue.cc:1035] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cff7321d643740eab7241f3549bea21b" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 44885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:29:14.620108 10503 consensus_queue.cc:1035] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb40556e6b444b44a4c61581ce6a4b06" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 33539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:29:14.629940 10511 mvcc.cc:205] Tried to move back new op lower bound from 7316877126111363072 to 7316877126003040256. Current Snapshot: MvccSnapshot[applied={T|T < 7316877126111363072}]
I20260810 07:29:14.630158 10512 mvcc.cc:205] Tried to move back new op lower bound from 7316877126111363072 to 7316877126003040256. Current Snapshot: MvccSnapshot[applied={T|T < 7316877126111363072}]
I20260810 07:29:14.636394 9711 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:29:14.636577 9711 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:29:14.639043 9711 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc99250996d0483bb935bec39a191e53: Sending DeleteTablet for 3 replicas of tablet 095ae8d57798438c87c1243e6f866b38
I20260810 07:29:14.639933 9929 tablet_service.cc:1514] Processing DeleteTablet for tablet 095ae8d57798438c87c1243e6f866b38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:29:14 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:54358
I20260810 07:29:14.639957 10149 tablet_service.cc:1514] Processing DeleteTablet for tablet 095ae8d57798438c87c1243e6f866b38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:29:14 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:60796
I20260810 07:29:14.639952 10364 tablet_service.cc:1514] Processing DeleteTablet for tablet 095ae8d57798438c87c1243e6f866b38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:29:14 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:34218
I20260810 07:29:14.640522 10531 tablet_replica.cc:330] T 095ae8d57798438c87c1243e6f866b38 P eb40556e6b444b44a4c61581ce6a4b06: stopping tablet replica
I20260810 07:29:14.640882 10531 raft_consensus.cc:2229] T 095ae8d57798438c87c1243e6f866b38 P eb40556e6b444b44a4c61581ce6a4b06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:29:14.641100 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 9822
I20260810 07:29:14.641248 10531 raft_consensus.cc:2258] T 095ae8d57798438c87c1243e6f866b38 P eb40556e6b444b44a4c61581ce6a4b06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:29:14.655957 10531 ts_tablet_manager.cc:1905] T 095ae8d57798438c87c1243e6f866b38 P eb40556e6b444b44a4c61581ce6a4b06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:29:14.659564 10531 ts_tablet_manager.cc:1918] T 095ae8d57798438c87c1243e6f866b38 P eb40556e6b444b44a4c61581ce6a4b06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:29:14.659694 10531 log.cc:1196] T 095ae8d57798438c87c1243e6f866b38 P eb40556e6b444b44a4c61581ce6a4b06: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-2/wal/wals/095ae8d57798438c87c1243e6f866b38
I20260810 07:29:14.660066 10531 ts_tablet_manager.cc:1939] T 095ae8d57798438c87c1243e6f866b38 P eb40556e6b444b44a4c61581ce6a4b06: Deleting consensus metadata
I20260810 07:29:14.661686 9687 catalog_manager.cc:4841] TS eb40556e6b444b44a4c61581ce6a4b06 (127.12.157.195:33539): tablet 095ae8d57798438c87c1243e6f866b38 (table test-table [id=b3a4063ebb8d4c2488c3d1e16bac3fe6]) successfully deleted
I20260810 07:29:14.665807 10530 tablet_replica.cc:330] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a: stopping tablet replica
I20260810 07:29:14.666191 10530 raft_consensus.cc:2229] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:29:14.666759 10530 raft_consensus.cc:2258] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:29:14.668280 10530 ts_tablet_manager.cc:1905] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:29:14.671142 10530 ts_tablet_manager.cc:1918] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:29:14.671277 10530 log.cc:1196] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786346694975122-12919-0/minicluster-data/ts-1/wal/wals/095ae8d57798438c87c1243e6f866b38
I20260810 07:29:14.671604 10530 ts_tablet_manager.cc:1939] T 095ae8d57798438c87c1243e6f866b38 P efe0b52cd1eb4f80a60a6baeeb5acc4a: Deleting consensus metadata
I20260810 07:29:14.672874 9688 catalog_manager.cc:4841] TS efe0b52cd1eb4f80a60a6baeeb5acc4a (127.12.157.194:37301): tablet 095ae8d57798438c87c1243e6f866b38 (table test-table [id=b3a4063ebb8d4c2488c3d1e16bac3fe6]) successfully deleted
W20260810 07:29:14.674232 9687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.157.193:44885 (error 108)
I20260810 07:29:14.675195 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 10050
I20260810 07:29:14.675474 10476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:14.675119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:44885 (local address 127.0.0.1:54364)
0810 07:29:14.675287 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:14.675369 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.193:44885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 07:29:14.685778 9687 catalog_manager.cc:4584] TS cff7321d643740eab7241f3549bea21b (127.12.157.193:44885): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 095ae8d57798438c87c1243e6f866b38: Network error: Client connection negotiation failed: client connection to 127.12.157.193:44885: connect: Connection refused (error 111)
I20260810 07:29:14.702073 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 10263
I20260810 07:29:14.714819 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 9659
2026-08-10T07:29:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2657 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T07:29:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:14Z Disabled control of system clock
I20260810 07:29:14.757851 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:44197
--webserver_interface=127.12.157.254
--webserver_port=0
--builtin_ntp_servers=127.12.157.212:40609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:44197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 07:29:14.879994 10537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:14.880306 10537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:14.880345 10537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:14.884850 10537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:14.884981 10537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:14.885001 10537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:14.885022 10537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:14.885038 10537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:14.890410 10537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:40609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:44197
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:44197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:14.891595 10537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:14.893119 10537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:14.902235 10547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:14.902262 10544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:14.902244 10543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:14.902616 10537 server_base.cc:1034] running on GCE node
I20260810 07:29:14.903213 10537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:14.904466 10537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:14.905674 10537 hybrid_clock.cc:648] HybridClock initialized: now 1786346954905660 us; error 27 us; skew 500 ppm
I20260810 07:29:14.907809 10537 webserver.cc:460] Webserver started at http://127.12.157.254:35323/ using document root <none> and password file <none>
I20260810 07:29:14.908481 10537 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:14.908586 10537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:14.908840 10537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:14.911368 10537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "74e68dbebe7d41ffbae31e3e79c9abf5"
format_stamp: "Formatted at 2026-08-10 07:29:14 on dist-test-slave-ffrd"
server_key: "16a5851aa14524b463dc2302baf6efa4"
server_key_iv: "5fa1c9c44d4de69320944201462f87ef"
server_key_version: "encryptionkey@0"
I20260810 07:29:14.912004 10537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "74e68dbebe7d41ffbae31e3e79c9abf5"
format_stamp: "Formatted at 2026-08-10 07:29:14 on dist-test-slave-ffrd"
server_key: "16a5851aa14524b463dc2302baf6efa4"
server_key_iv: "5fa1c9c44d4de69320944201462f87ef"
server_key_version: "encryptionkey@0"
I20260810 07:29:14.916114 10537 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 07:29:14.919535 10559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:14.921000 10537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 07:29:14.921164 10537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "74e68dbebe7d41ffbae31e3e79c9abf5"
format_stamp: "Formatted at 2026-08-10 07:29:14 on dist-test-slave-ffrd"
server_key: "16a5851aa14524b463dc2302baf6efa4"
server_key_iv: "5fa1c9c44d4de69320944201462f87ef"
server_key_version: "encryptionkey@0"
I20260810 07:29:14.921304 10537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:14.949894 10537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:14.950685 10537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:14.950917 10537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:14.957930 10537 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:44197
I20260810 07:29:14.957968 10645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:44197 every 8 connection(s)
I20260810 07:29:14.958879 10537 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:14.961993 10648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:14.964998 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 10537
I20260810 07:29:14.965138 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:29:14.965428 12919 external_mini_cluster.cc:1428] Setting key 3c8faf308b6f0e9e49f6092890dcc58e
I20260810 07:29:14.968664 10648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74e68dbebe7d41ffbae31e3e79c9abf5: Bootstrap starting.
I20260810 07:29:14.970923 10648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74e68dbebe7d41ffbae31e3e79c9abf5: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:14.971678 10648 log.cc:824] T 00000000000000000000000000000000 P 74e68dbebe7d41ffbae31e3e79c9abf5: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:14.973827 10648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74e68dbebe7d41ffbae31e3e79c9abf5: No bootstrap required, opened a new log
I20260810 07:29:14.975936 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:14.966969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44474 (local address 127.12.157.254:44197)
0810 07:29:14.967784 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:14.967814 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:14.967875 (+ 61us) server_negotiation.cc:407] Connection header received
0810 07:29:14.968881 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:14.968901 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:14.969359 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:14.969695 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:14.970154 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.971203 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.972176 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:14.972658 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:14.972865 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:14.973965 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:14.973971 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:14.973989 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:14.974403 (+ 414us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:14.974462 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:14.974483 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:14.974600 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:14.974752 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 07:29:14.975237 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":140,"threads_started":1}
I20260810 07:29:14.977996 10648 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74e68dbebe7d41ffbae31e3e79c9abf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e68dbebe7d41ffbae31e3e79c9abf5" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44197 } }
I20260810 07:29:14.978298 10648 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74e68dbebe7d41ffbae31e3e79c9abf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:14.978359 10648 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74e68dbebe7d41ffbae31e3e79c9abf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74e68dbebe7d41ffbae31e3e79c9abf5, State: Initialized, Role: FOLLOWER
I20260810 07:29:14.978798 10648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74e68dbebe7d41ffbae31e3e79c9abf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e68dbebe7d41ffbae31e3e79c9abf5" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44197 } }
I20260810 07:29:14.978927 10648 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74e68dbebe7d41ffbae31e3e79c9abf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:29:14.979008 10648 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74e68dbebe7d41ffbae31e3e79c9abf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:29:14.979127 10648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74e68dbebe7d41ffbae31e3e79c9abf5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:14.980085 10648 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74e68dbebe7d41ffbae31e3e79c9abf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e68dbebe7d41ffbae31e3e79c9abf5" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44197 } }
I20260810 07:29:14.980443 10648 leader_election.cc:302] T 00000000000000000000000000000000 P 74e68dbebe7d41ffbae31e3e79c9abf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74e68dbebe7d41ffbae31e3e79c9abf5; no voters:
I20260810 07:29:14.980808 10648 leader_election.cc:288] T 00000000000000000000000000000000 P 74e68dbebe7d41ffbae31e3e79c9abf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:29:14.981122 10655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74e68dbebe7d41ffbae31e3e79c9abf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:14.981344 10655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74e68dbebe7d41ffbae31e3e79c9abf5 [term 1 LEADER]: Becoming Leader. State: Replica: 74e68dbebe7d41ffbae31e3e79c9abf5, State: Running, Role: LEADER
I20260810 07:29:14.981659 10655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74e68dbebe7d41ffbae31e3e79c9abf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e68dbebe7d41ffbae31e3e79c9abf5" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44197 } }
I20260810 07:29:14.982120 10648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74e68dbebe7d41ffbae31e3e79c9abf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:14.983611 10658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74e68dbebe7d41ffbae31e3e79c9abf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74e68dbebe7d41ffbae31e3e79c9abf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e68dbebe7d41ffbae31e3e79c9abf5" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44197 } } }
I20260810 07:29:14.983767 10658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74e68dbebe7d41ffbae31e3e79c9abf5 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:14.983623 10659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74e68dbebe7d41ffbae31e3e79c9abf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74e68dbebe7d41ffbae31e3e79c9abf5. Latest consensus state: current_term: 1 leader_uuid: "74e68dbebe7d41ffbae31e3e79c9abf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74e68dbebe7d41ffbae31e3e79c9abf5" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44197 } } }
I20260810 07:29:14.983925 10659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74e68dbebe7d41ffbae31e3e79c9abf5 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:14.984623 10672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:14.987594 10672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:14.993650 10672 catalog_manager.cc:1347] Generated new cluster ID: 0fdd82c43fa64ebdad0ce1b03a8ce761
I20260810 07:29:14.993738 10672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:15.002612 10672 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:15.003911 10672 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:15.013866 10672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74e68dbebe7d41ffbae31e3e79c9abf5: Generated new TSK 0
I20260810 07:29:15.014739 10672 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:29:15.018563 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:0
--local_ip_for_outbound_sockets=127.12.157.193
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:44197
--builtin_ntp_servers=127.12.157.212:40609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 07:29:15.141403 10692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:15.141700 10692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:15.141741 10692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:15.146396 10692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:15.146487 10692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:15.146551 10692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:29:15.151993 10692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:40609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:44197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:15.153239 10692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:15.154699 10692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:15.165328 10704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:15.165325 10701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:15.165320 10702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:15.165565 10692 server_base.cc:1034] running on GCE node
I20260810 07:29:15.166103 10692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:15.166819 10692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:15.168030 10692 hybrid_clock.cc:648] HybridClock initialized: now 1786346955167967 us; error 47 us; skew 500 ppm
I20260810 07:29:15.170186 10692 webserver.cc:460] Webserver started at http://127.12.157.193:35767/ using document root <none> and password file <none>
I20260810 07:29:15.170761 10692 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:15.170835 10692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:15.171011 10692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:15.173307 10692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-0/data/instance:
uuid: "c4dd8966870f4b1a98474f0f6a6aa868"
format_stamp: "Formatted at 2026-08-10 07:29:15 on dist-test-slave-ffrd"
server_key: "e51281976b3cd0f6e95b04b1ca0ff9a2"
server_key_iv: "762109f272c9ffb9ce757d35b387b27f"
server_key_version: "encryptionkey@0"
I20260810 07:29:15.173890 10692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance:
uuid: "c4dd8966870f4b1a98474f0f6a6aa868"
format_stamp: "Formatted at 2026-08-10 07:29:15 on dist-test-slave-ffrd"
server_key: "e51281976b3cd0f6e95b04b1ca0ff9a2"
server_key_iv: "762109f272c9ffb9ce757d35b387b27f"
server_key_version: "encryptionkey@0"
I20260810 07:29:15.177981 10692 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 07:29:15.180671 10718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:15.182133 10692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 07:29:15.182245 10692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "c4dd8966870f4b1a98474f0f6a6aa868"
format_stamp: "Formatted at 2026-08-10 07:29:15 on dist-test-slave-ffrd"
server_key: "e51281976b3cd0f6e95b04b1ca0ff9a2"
server_key_iv: "762109f272c9ffb9ce757d35b387b27f"
server_key_version: "encryptionkey@0"
I20260810 07:29:15.182343 10692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:15.203130 10692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:15.203893 10692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:15.204111 10692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:15.205026 10692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:29:15.206077 10692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:29:15.206128 10692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:15.206202 10692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:29:15.206233 10692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:15.215798 10692 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:39267
I20260810 07:29:15.215817 10893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:39267 every 8 connection(s)
I20260810 07:29:15.216826 10692 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:29:15.226161 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:15.218723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:49771 (local address 127.12.157.254:44197)
0810 07:29:15.218902 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:15.218906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:15.219940 (+ 1034us) server_negotiation.cc:407] Connection header received
0810 07:29:15.221335 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:15.221338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:15.221388 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:15.221475 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:15.222466 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.222998 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.224145 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.224368 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.224449 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:15.225677 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:15.225680 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:15.225681 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:15.225695 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:15.225714 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:15.225722 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:15.225783 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:15.225979 (+ 196us) server_negotiation.cc:299] Negotiation successful
0810 07:29:15.226043 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 07:29:15.227018 10899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:15.219103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:44197 (local address 127.12.157.193:49771)
0810 07:29:15.219794 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:15.219833 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:15.221104 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:15.221685 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:15.221692 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:15.221773 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:15.222273 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:15.222287 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.223118 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.223120 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:15.224030 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:15.224042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.224512 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.224516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:15.224667 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:15.225411 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:29:15.225412 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:15.225463 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:29:15.225532 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:15.225799 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:15.225806 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:15.225811 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:29:15.225934 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 07:29:15.226187 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":117,"threads_started":1}
I20260810 07:29:15.227079 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 10692
I20260810 07:29:15.227203 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance
I20260810 07:29:15.227509 12919 external_mini_cluster.cc:1428] Setting key cf38abbd4116fadcc3712e9be025d388
I20260810 07:29:15.228385 10894 heartbeater.cc:347] Connected to a master server at 127.12.157.254:44197
I20260810 07:29:15.228681 10894 heartbeater.cc:464] Registering TS with master...
I20260810 07:29:15.229271 10894 heartbeater.cc:511] Master 127.12.157.254:44197 requested a full tablet report, sending...
I20260810 07:29:15.229503 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:0
--local_ip_for_outbound_sockets=127.12.157.194
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:44197
--builtin_ntp_servers=127.12.157.212:40609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 07:29:15.230952 10590 ts_manager.cc:194] Registered new tserver with Master: c4dd8966870f4b1a98474f0f6a6aa868 (127.12.157.193:39267)
I20260810 07:29:15.232234 10590 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.157.193:49771
W20260810 07:29:15.346338 10900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:15.346652 10900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:15.346720 10900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:15.351392 10900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:15.351485 10900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:15.351547 10900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:29:15.356729 10900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:40609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:44197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:15.358044 10900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:15.359503 10900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:15.370173 10915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:15.370345 10900 server_base.cc:1034] running on GCE node
W20260810 07:29:15.370193 10909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:15.370579 10910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:15.371040 10900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:15.371747 10900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:15.373026 10900 hybrid_clock.cc:648] HybridClock initialized: now 1786346955372979 us; error 52 us; skew 500 ppm
I20260810 07:29:15.375030 10900 webserver.cc:460] Webserver started at http://127.12.157.194:38137/ using document root <none> and password file <none>
I20260810 07:29:15.375690 10900 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:15.375787 10900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:15.376013 10900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:15.378247 10900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-1/data/instance:
uuid: "b276a048c139457286627db8cbb12a36"
format_stamp: "Formatted at 2026-08-10 07:29:15 on dist-test-slave-ffrd"
server_key: "d683f476664eddc261fbc5a04963887a"
server_key_iv: "5b2ec5c2388e956209d536833523342c"
server_key_version: "encryptionkey@0"
I20260810 07:29:15.378846 10900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance:
uuid: "b276a048c139457286627db8cbb12a36"
format_stamp: "Formatted at 2026-08-10 07:29:15 on dist-test-slave-ffrd"
server_key: "d683f476664eddc261fbc5a04963887a"
server_key_iv: "5b2ec5c2388e956209d536833523342c"
server_key_version: "encryptionkey@0"
I20260810 07:29:15.382792 10900 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 07:29:15.385900 10921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:15.387526 10900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 07:29:15.387696 10900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "b276a048c139457286627db8cbb12a36"
format_stamp: "Formatted at 2026-08-10 07:29:15 on dist-test-slave-ffrd"
server_key: "d683f476664eddc261fbc5a04963887a"
server_key_iv: "5b2ec5c2388e956209d536833523342c"
server_key_version: "encryptionkey@0"
I20260810 07:29:15.387827 10900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:15.424432 10900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:15.425565 10900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:15.425806 10900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:15.426549 10900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:29:15.427723 10900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:29:15.427780 10900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:15.427863 10900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:29:15.427906 10900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:15.437391 10900 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:45109
I20260810 07:29:15.437495 11121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:45109 every 8 connection(s)
I20260810 07:29:15.438453 10900 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:29:15.438845 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 10900
I20260810 07:29:15.439013 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance
I20260810 07:29:15.439368 12919 external_mini_cluster.cc:1428] Setting key fca9de5c4c64f7e84bd1ef8a6349a250
I20260810 07:29:15.441522 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:0
--local_ip_for_outbound_sockets=127.12.157.195
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:44197
--builtin_ntp_servers=127.12.157.212:40609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 07:29:15.449708 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:15.440392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:49423 (local address 127.12.157.254:44197)
0810 07:29:15.440547 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:15.440552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:15.441914 (+ 1362us) server_negotiation.cc:407] Connection header received
0810 07:29:15.443341 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:15.443347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:15.443416 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:15.443517 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:15.444787 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.445470 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.447394 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.447611 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.447693 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:15.449149 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:15.449152 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:15.449154 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:15.449167 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:15.449189 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:15.449203 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:15.449290 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:15.449522 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 07:29:15.449586 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 07:29:15.450995 11130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:15.440976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:44197 (local address 127.12.157.194:49423)
0810 07:29:15.441739 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:15.441793 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:15.443086 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:15.443736 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:15.443748 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:15.443863 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:15.444616 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:15.444640 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.445609 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.445616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:15.447240 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:15.447255 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.447788 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.447795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:15.448049 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:15.448779 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:29:15.448782 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:15.448843 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:29:15.448967 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:15.449312 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:15.449323 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:15.449331 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 07:29:15.449502 (+ 171us) client_negotiation.cc:240] Negotiation successful
0810 07:29:15.449848 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":134,"threads_started":1}
I20260810 07:29:15.452850 11122 heartbeater.cc:347] Connected to a master server at 127.12.157.254:44197
I20260810 07:29:15.453161 11122 heartbeater.cc:464] Registering TS with master...
I20260810 07:29:15.453943 11122 heartbeater.cc:511] Master 127.12.157.254:44197 requested a full tablet report, sending...
I20260810 07:29:15.455273 10590 ts_manager.cc:194] Registered new tserver with Master: b276a048c139457286627db8cbb12a36 (127.12.157.194:45109)
I20260810 07:29:15.456136 10590 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.157.194:49423
W20260810 07:29:15.565400 11134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:15.565711 11134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:15.565780 11134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:15.569921 11134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:15.569972 11134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:15.570025 11134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:29:15.574705 11134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:40609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:44197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:15.575809 11134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:15.577361 11134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:15.587075 11141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:15.587102 11142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:15.587281 11134 server_base.cc:1034] running on GCE node
W20260810 07:29:15.587121 11144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:15.587769 11134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:15.588460 11134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:15.589655 11134 hybrid_clock.cc:648] HybridClock initialized: now 1786346955589624 us; error 42 us; skew 500 ppm
I20260810 07:29:15.591662 11134 webserver.cc:460] Webserver started at http://127.12.157.195:45843/ using document root <none> and password file <none>
I20260810 07:29:15.592306 11134 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:15.592401 11134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:15.592638 11134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:15.594767 11134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-2/data/instance:
uuid: "22cd1f41716a47c4ab829b573c28f658"
format_stamp: "Formatted at 2026-08-10 07:29:15 on dist-test-slave-ffrd"
server_key: "e8952c89dbdadb22dc368f3bb00c13bb"
server_key_iv: "bfeaf5d8eb7ed048ffaa5c54fa4ae265"
server_key_version: "encryptionkey@0"
I20260810 07:29:15.595356 11134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance:
uuid: "22cd1f41716a47c4ab829b573c28f658"
format_stamp: "Formatted at 2026-08-10 07:29:15 on dist-test-slave-ffrd"
server_key: "e8952c89dbdadb22dc368f3bb00c13bb"
server_key_iv: "bfeaf5d8eb7ed048ffaa5c54fa4ae265"
server_key_version: "encryptionkey@0"
I20260810 07:29:15.599073 11134 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 07:29:15.601552 11152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:15.602730 11134 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:29:15.602885 11134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "22cd1f41716a47c4ab829b573c28f658"
format_stamp: "Formatted at 2026-08-10 07:29:15 on dist-test-slave-ffrd"
server_key: "e8952c89dbdadb22dc368f3bb00c13bb"
server_key_iv: "bfeaf5d8eb7ed048ffaa5c54fa4ae265"
server_key_version: "encryptionkey@0"
I20260810 07:29:15.603004 11134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:15.617035 11134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:15.617704 11134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:15.617914 11134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:15.618507 11134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:29:15.619474 11134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:29:15.619560 11134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:15.619622 11134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:29:15.619659 11134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:15.629562 11134 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:34785
I20260810 07:29:15.629586 11346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:34785 every 8 connection(s)
I20260810 07:29:15.630538 11134 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:29:15.640177 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 11134
I20260810 07:29:15.640256 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:15.632543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:46749 (local address 127.12.157.254:44197)
0810 07:29:15.632742 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:15.632748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:15.633869 (+ 1121us) server_negotiation.cc:407] Connection header received
0810 07:29:15.635236 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:15.635239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:15.635288 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:15.635381 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:15.636317 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.636875 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.638112 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.638358 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.638462 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:15.639699 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:15.639702 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:15.639704 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:15.639718 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:15.639739 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:15.639746 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:15.639818 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:15.640007 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 07:29:15.640082 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 07:29:15.640292 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance
I20260810 07:29:15.640592 12919 external_mini_cluster.cc:1428] Setting key c2bf06a3f1f0f108f61ca5119a263991
I20260810 07:29:15.641254 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:15.633004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:44197 (local address 127.12.157.195:46749)
0810 07:29:15.633699 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:15.633742 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:15.635017 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:15.635548 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:15.635554 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:15.635634 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:15.636169 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:15.636182 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.637064 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.637067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:15.637981 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:15.637991 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.638545 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.638549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:15.638714 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:15.639419 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:29:15.639420 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:15.639461 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:29:15.639527 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:15.639849 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:15.639857 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:15.639862 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 07:29:15.639964 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 07:29:15.640192 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":132,"threads_started":1}
I20260810 07:29:15.642494 11347 heartbeater.cc:347] Connected to a master server at 127.12.157.254:44197
I20260810 07:29:15.642735 11347 heartbeater.cc:464] Registering TS with master...
I20260810 07:29:15.643299 11347 heartbeater.cc:511] Master 127.12.157.254:44197 requested a full tablet report, sending...
I20260810 07:29:15.644349 10590 ts_manager.cc:194] Registered new tserver with Master: 22cd1f41716a47c4ab829b573c28f658 (127.12.157.195:34785)
I20260810 07:29:15.645040 10590 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.157.195:46749
I20260810 07:29:15.654929 12919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:29:15.661032 10653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:15.657059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44490 (local address 127.12.157.254:44197)
0810 07:29:15.657208 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:15.657214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:15.657411 (+ 197us) server_negotiation.cc:407] Connection header received
0810 07:29:15.657483 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:15.657485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:15.657530 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:15.657605 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:15.657895 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.658438 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.659347 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.659570 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.659648 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:15.660539 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:15.660542 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:15.660544 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:15.660554 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:15.660571 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:15.660578 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:15.660660 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:15.660786 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 07:29:15.660843 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 07:29:15.664331 10590 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:29:15.667434 10590 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:29:15.683035 11368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:15.677665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39264 (local address 127.12.157.195:34785)
0810 07:29:15.678038 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:15.678050 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:15.678069 (+ 19us) server_negotiation.cc:407] Connection header received
0810 07:29:15.678134 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:15.678138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:15.678282 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:15.678797 (+ 515us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:15.678957 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.680092 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.681627 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.682288 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.682423 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:15.682613 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:15.682783 (+ 170us) server_negotiation.cc:299] Negotiation successful
0810 07:29:15.682889 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":146,"threads_started":1}
I20260810 07:29:15.683027 11366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:15.677334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:34785 (local address 127.0.0.1:39264)
0810 07:29:15.677590 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:15.677602 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:15.677684 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:15.678448 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:15.678454 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:15.678476 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:15.678638 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:15.678646 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.680226 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.680231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:15.681488 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:15.681501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.682418 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.682421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:15.682560 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:15.682599 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 07:29:15.682719 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 07:29:15.682862 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":128,"thread_start_us":110,"threads_started":1}
I20260810 07:29:15.683207 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:15.676543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:39267 (local address 127.0.0.1:42100)
0810 07:29:15.676928 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:15.677018 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:15.677150 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:15.678563 (+ 1413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:15.678565 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:15.678582 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:15.678724 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:15.678734 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.680178 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.680182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:15.681428 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:15.681437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.682444 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.682447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:15.682838 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:15.682863 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 07:29:15.682964 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 07:29:15.683039 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":329,"thread_start_us":95,"threads_started":1}
I20260810 07:29:15.683297 11364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:15.676664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42100 (local address 127.12.157.193:39267)
0810 07:29:15.677891 (+ 1227us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:15.677905 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:15.677932 (+ 27us) server_negotiation.cc:407] Connection header received
0810 07:29:15.678012 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:15.678017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:15.678224 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:15.678403 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:15.678852 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.680074 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.681552 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.682291 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.682508 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:15.682678 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:15.682999 (+ 321us) server_negotiation.cc:299] Negotiation successful
0810 07:29:15.683138 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1110,"thread_start_us":131,"threads_started":1}
I20260810 07:29:15.683825 11365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:15.677036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:45109 (local address 127.0.0.1:49942)
0810 07:29:15.677312 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:15.677325 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:15.677416 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:15.678645 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:15.678648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:15.678670 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:15.678795 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:15.678802 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.679799 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.679805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:15.681310 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:15.681323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.682441 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.682446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:15.683528 (+ 1082us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:15.683554 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 07:29:15.683627 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 07:29:15.683701 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":985,"thread_start_us":99,"threads_started":1}
I20260810 07:29:15.684450 11367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:15.677481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49942 (local address 127.12.157.194:45109)
0810 07:29:15.678147 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:15.678160 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:15.678186 (+ 26us) server_negotiation.cc:407] Connection header received
0810 07:29:15.678267 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:15.678276 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:15.678489 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:15.678666 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:15.678928 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.679680 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.681465 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.682291 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.682551 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:15.682691 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:15.684151 (+ 1460us) server_negotiation.cc:299] Negotiation successful
0810 07:29:15.684259 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":111,"threads_started":1}
I20260810 07:29:15.686720 10786 tablet_service.cc:1467] Processing CreateTablet for tablet e26e10ec787643acb8185fd841ccba0e (DEFAULT_TABLE table=test-table [id=1773ac2cbd3f467d86586319a821f115]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:29:15.686723 11240 tablet_service.cc:1467] Processing CreateTablet for tablet e26e10ec787643acb8185fd841ccba0e (DEFAULT_TABLE table=test-table [id=1773ac2cbd3f467d86586319a821f115]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:29:15.687023 11004 tablet_service.cc:1467] Processing CreateTablet for tablet e26e10ec787643acb8185fd841ccba0e (DEFAULT_TABLE table=test-table [id=1773ac2cbd3f467d86586319a821f115]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:29:15.687901 10786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e26e10ec787643acb8185fd841ccba0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:15.687901 11240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e26e10ec787643acb8185fd841ccba0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:15.688082 11004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e26e10ec787643acb8185fd841ccba0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:15.695119 11369 tablet_bootstrap.cc:492] T e26e10ec787643acb8185fd841ccba0e P b276a048c139457286627db8cbb12a36: Bootstrap starting.
I20260810 07:29:15.696081 11371 tablet_bootstrap.cc:492] T e26e10ec787643acb8185fd841ccba0e P c4dd8966870f4b1a98474f0f6a6aa868: Bootstrap starting.
I20260810 07:29:15.697346 11373 tablet_bootstrap.cc:492] T e26e10ec787643acb8185fd841ccba0e P 22cd1f41716a47c4ab829b573c28f658: Bootstrap starting.
I20260810 07:29:15.697815 11369 tablet_bootstrap.cc:654] T e26e10ec787643acb8185fd841ccba0e P b276a048c139457286627db8cbb12a36: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:15.698158 11371 tablet_bootstrap.cc:654] T e26e10ec787643acb8185fd841ccba0e P c4dd8966870f4b1a98474f0f6a6aa868: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:15.699010 11369 log.cc:824] T e26e10ec787643acb8185fd841ccba0e P b276a048c139457286627db8cbb12a36: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:15.699033 11371 log.cc:824] T e26e10ec787643acb8185fd841ccba0e P c4dd8966870f4b1a98474f0f6a6aa868: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:15.699556 11373 tablet_bootstrap.cc:654] T e26e10ec787643acb8185fd841ccba0e P 22cd1f41716a47c4ab829b573c28f658: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:15.700484 11373 log.cc:824] T e26e10ec787643acb8185fd841ccba0e P 22cd1f41716a47c4ab829b573c28f658: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:15.701638 11369 tablet_bootstrap.cc:492] T e26e10ec787643acb8185fd841ccba0e P b276a048c139457286627db8cbb12a36: No bootstrap required, opened a new log
I20260810 07:29:15.701957 11369 ts_tablet_manager.cc:1397] T e26e10ec787643acb8185fd841ccba0e P b276a048c139457286627db8cbb12a36: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 07:29:15.702351 11373 tablet_bootstrap.cc:492] T e26e10ec787643acb8185fd841ccba0e P 22cd1f41716a47c4ab829b573c28f658: No bootstrap required, opened a new log
I20260810 07:29:15.702417 11371 tablet_bootstrap.cc:492] T e26e10ec787643acb8185fd841ccba0e P c4dd8966870f4b1a98474f0f6a6aa868: No bootstrap required, opened a new log
I20260810 07:29:15.702584 11373 ts_tablet_manager.cc:1397] T e26e10ec787643acb8185fd841ccba0e P 22cd1f41716a47c4ab829b573c28f658: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 07:29:15.702678 11371 ts_tablet_manager.cc:1397] T e26e10ec787643acb8185fd841ccba0e P c4dd8966870f4b1a98474f0f6a6aa868: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.002s
I20260810 07:29:15.706166 11373 raft_consensus.cc:348] T e26e10ec787643acb8185fd841ccba0e P 22cd1f41716a47c4ab829b573c28f658 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4dd8966870f4b1a98474f0f6a6aa868" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39267 } } peers { permanent_uuid: "22cd1f41716a47c4ab829b573c28f658" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34785 } } peers { permanent_uuid: "b276a048c139457286627db8cbb12a36" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 45109 } }
I20260810 07:29:15.706692 11373 raft_consensus.cc:374] T e26e10ec787643acb8185fd841ccba0e P 22cd1f41716a47c4ab829b573c28f658 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:15.706535 11369 raft_consensus.cc:348] T e26e10ec787643acb8185fd841ccba0e P b276a048c139457286627db8cbb12a36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4dd8966870f4b1a98474f0f6a6aa868" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39267 } } peers { permanent_uuid: "22cd1f41716a47c4ab829b573c28f658" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34785 } } peers { permanent_uuid: "b276a048c139457286627db8cbb12a36" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 45109 } }
I20260810 07:29:15.706789 11373 raft_consensus.cc:729] T e26e10ec787643acb8185fd841ccba0e P 22cd1f41716a47c4ab829b573c28f658 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22cd1f41716a47c4ab829b573c28f658, State: Initialized, Role: FOLLOWER
I20260810 07:29:15.706905 11369 raft_consensus.cc:374] T e26e10ec787643acb8185fd841ccba0e P b276a048c139457286627db8cbb12a36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:15.706988 11369 raft_consensus.cc:729] T e26e10ec787643acb8185fd841ccba0e P b276a048c139457286627db8cbb12a36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b276a048c139457286627db8cbb12a36, State: Initialized, Role: FOLLOWER
I20260810 07:29:15.707041 11371 raft_consensus.cc:348] T e26e10ec787643acb8185fd841ccba0e P c4dd8966870f4b1a98474f0f6a6aa868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4dd8966870f4b1a98474f0f6a6aa868" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39267 } } peers { permanent_uuid: "22cd1f41716a47c4ab829b573c28f658" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34785 } } peers { permanent_uuid: "b276a048c139457286627db8cbb12a36" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 45109 } }
I20260810 07:29:15.707347 11371 raft_consensus.cc:374] T e26e10ec787643acb8185fd841ccba0e P c4dd8966870f4b1a98474f0f6a6aa868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:15.707407 11371 raft_consensus.cc:729] T e26e10ec787643acb8185fd841ccba0e P c4dd8966870f4b1a98474f0f6a6aa868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4dd8966870f4b1a98474f0f6a6aa868, State: Initialized, Role: FOLLOWER
I20260810 07:29:15.707342 11373 consensus_queue.cc:260] T e26e10ec787643acb8185fd841ccba0e P 22cd1f41716a47c4ab829b573c28f658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4dd8966870f4b1a98474f0f6a6aa868" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39267 } } peers { permanent_uuid: "22cd1f41716a47c4ab829b573c28f658" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34785 } } peers { permanent_uuid: "b276a048c139457286627db8cbb12a36" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 45109 } }
I20260810 07:29:15.707484 11369 consensus_queue.cc:260] T e26e10ec787643acb8185fd841ccba0e P b276a048c139457286627db8cbb12a36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4dd8966870f4b1a98474f0f6a6aa868" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39267 } } peers { permanent_uuid: "22cd1f41716a47c4ab829b573c28f658" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34785 } } peers { permanent_uuid: "b276a048c139457286627db8cbb12a36" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 45109 } }
I20260810 07:29:15.707859 11371 consensus_queue.cc:260] T e26e10ec787643acb8185fd841ccba0e P c4dd8966870f4b1a98474f0f6a6aa868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4dd8966870f4b1a98474f0f6a6aa868" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39267 } } peers { permanent_uuid: "22cd1f41716a47c4ab829b573c28f658" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34785 } } peers { permanent_uuid: "b276a048c139457286627db8cbb12a36" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 45109 } }
I20260810 07:29:15.708094 11347 heartbeater.cc:503] Master 127.12.157.254:44197 was elected leader, sending a full tablet report...
I20260810 07:29:15.708464 11373 ts_tablet_manager.cc:1428] T e26e10ec787643acb8185fd841ccba0e P 22cd1f41716a47c4ab829b573c28f658: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 07:29:15.708550 11369 ts_tablet_manager.cc:1428] T e26e10ec787643acb8185fd841ccba0e P b276a048c139457286627db8cbb12a36: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 07:29:15.708786 11371 ts_tablet_manager.cc:1428] T e26e10ec787643acb8185fd841ccba0e P c4dd8966870f4b1a98474f0f6a6aa868: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260810 07:29:15.708890 11122 heartbeater.cc:503] Master 127.12.157.254:44197 was elected leader, sending a full tablet report...
I20260810 07:29:15.709352 10894 heartbeater.cc:503] Master 127.12.157.254:44197 was elected leader, sending a full tablet report...
W20260810 07:29:15.718026 10897 tablet.cc:2369] T e26e10ec787643acb8185fd841ccba0e P c4dd8966870f4b1a98474f0f6a6aa868: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:29:15.763165 11384 raft_consensus.cc:482] T e26e10ec787643acb8185fd841ccba0e P c4dd8966870f4b1a98474f0f6a6aa868 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:29:15.763430 11384 raft_consensus.cc:504] T e26e10ec787643acb8185fd841ccba0e P c4dd8966870f4b1a98474f0f6a6aa868 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4dd8966870f4b1a98474f0f6a6aa868" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39267 } } peers { permanent_uuid: "22cd1f41716a47c4ab829b573c28f658" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34785 } } peers { permanent_uuid: "b276a048c139457286627db8cbb12a36" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 45109 } }
I20260810 07:29:15.764628 11384 leader_election.cc:288] T e26e10ec787643acb8185fd841ccba0e P c4dd8966870f4b1a98474f0f6a6aa868 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22cd1f41716a47c4ab829b573c28f658 (127.12.157.195:34785), b276a048c139457286627db8cbb12a36 (127.12.157.194:45109)
I20260810 07:29:15.769116 11368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:15.765132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:53401 (local address 127.12.157.195:34785)
0810 07:29:15.765266 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:15.765273 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:15.765478 (+ 205us) server_negotiation.cc:407] Connection header received
0810 07:29:15.765600 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:15.765603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:15.765669 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:15.765748 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:15.766034 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.766523 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.767848 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.768503 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.768639 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:15.768711 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:15.768874 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 07:29:15.768970 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 07:29:15.769163 11386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:15.764876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:34785 (local address 127.12.157.193:53401)
0810 07:29:15.765364 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:15.765379 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:15.765496 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:15.765781 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:15.765784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:15.765799 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:15.765902 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:15.765907 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.766624 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.766628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:15.767703 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:15.767722 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.768636 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.768643 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:15.768747 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:15.768771 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 07:29:15.768838 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 07:29:15.768979 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":53,"thread_start_us":129,"threads_started":1}
I20260810 07:29:15.769536 11367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:15.765053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:33667 (local address 127.12.157.194:45109)
0810 07:29:15.765275 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:15.765279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:15.765377 (+ 98us) server_negotiation.cc:407] Connection header received
0810 07:29:15.765512 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:15.765518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:15.765582 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:15.765649 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:15.765998 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.766640 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.767936 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.768611 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.768723 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:15.768770 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:15.769241 (+ 471us) server_negotiation.cc:299] Negotiation successful
0810 07:29:15.769362 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260810 07:29:15.769893 11385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:15.764876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:45109 (local address 127.12.157.193:33667)
0810 07:29:15.765265 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:15.765281 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:15.765397 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:15.765684 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:15.765689 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:15.765710 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:15.765875 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:15.765882 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.766758 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.766761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:15.767797 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:15.767811 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.768840 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.768844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:15.769034 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:15.769058 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 07:29:15.769678 (+ 620us) client_negotiation.cc:240] Negotiation successful
0810 07:29:15.769757 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":28,"thread_start_us":114,"threads_started":1}
I20260810 07:29:15.769949 11276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e26e10ec787643acb8185fd841ccba0e" candidate_uuid: "c4dd8966870f4b1a98474f0f6a6aa868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22cd1f41716a47c4ab829b573c28f658" is_pre_election: true
I20260810 07:29:15.770308 11276 raft_consensus.cc:2454] T e26e10ec787643acb8185fd841ccba0e P 22cd1f41716a47c4ab829b573c28f658 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4dd8966870f4b1a98474f0f6a6aa868 in term 0.
I20260810 07:29:15.770893 10722 leader_election.cc:302] T e26e10ec787643acb8185fd841ccba0e P c4dd8966870f4b1a98474f0f6a6aa868 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22cd1f41716a47c4ab829b573c28f658, c4dd8966870f4b1a98474f0f6a6aa868; no voters:
I20260810 07:29:15.771112 11044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e26e10ec787643acb8185fd841ccba0e" candidate_uuid: "c4dd8966870f4b1a98474f0f6a6aa868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276a048c139457286627db8cbb12a36" is_pre_election: true
I20260810 07:29:15.771179 11384 raft_consensus.cc:2781] T e26e10ec787643acb8185fd841ccba0e P c4dd8966870f4b1a98474f0f6a6aa868 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:29:15.771288 11384 raft_consensus.cc:482] T e26e10ec787643acb8185fd841ccba0e P c4dd8966870f4b1a98474f0f6a6aa868 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:29:15.771347 11384 raft_consensus.cc:3037] T e26e10ec787643acb8185fd841ccba0e P c4dd8966870f4b1a98474f0f6a6aa868 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:15.771421 11044 raft_consensus.cc:2454] T e26e10ec787643acb8185fd841ccba0e P b276a048c139457286627db8cbb12a36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4dd8966870f4b1a98474f0f6a6aa868 in term 0.
I20260810 07:29:15.772490 11384 raft_consensus.cc:504] T e26e10ec787643acb8185fd841ccba0e P c4dd8966870f4b1a98474f0f6a6aa868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4dd8966870f4b1a98474f0f6a6aa868" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39267 } } peers { permanent_uuid: "22cd1f41716a47c4ab829b573c28f658" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34785 } } peers { permanent_uuid: "b276a048c139457286627db8cbb12a36" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 45109 } }
I20260810 07:29:15.772876 11384 leader_election.cc:288] T e26e10ec787643acb8185fd841ccba0e P c4dd8966870f4b1a98474f0f6a6aa868 [CANDIDATE]: Term 1 election: Requested vote from peers 22cd1f41716a47c4ab829b573c28f658 (127.12.157.195:34785), b276a048c139457286627db8cbb12a36 (127.12.157.194:45109)
I20260810 07:29:15.773242 11044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e26e10ec787643acb8185fd841ccba0e" candidate_uuid: "c4dd8966870f4b1a98474f0f6a6aa868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b276a048c139457286627db8cbb12a36"
I20260810 07:29:15.773376 11044 raft_consensus.cc:3037] T e26e10ec787643acb8185fd841ccba0e P b276a048c139457286627db8cbb12a36 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:15.773432 11276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e26e10ec787643acb8185fd841ccba0e" candidate_uuid: "c4dd8966870f4b1a98474f0f6a6aa868" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22cd1f41716a47c4ab829b573c28f658"
I20260810 07:29:15.773562 11276 raft_consensus.cc:3037] T e26e10ec787643acb8185fd841ccba0e P 22cd1f41716a47c4ab829b573c28f658 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:15.774755 11044 raft_consensus.cc:2454] T e26e10ec787643acb8185fd841ccba0e P b276a048c139457286627db8cbb12a36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4dd8966870f4b1a98474f0f6a6aa868 in term 1.
I20260810 07:29:15.774816 11276 raft_consensus.cc:2454] T e26e10ec787643acb8185fd841ccba0e P 22cd1f41716a47c4ab829b573c28f658 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4dd8966870f4b1a98474f0f6a6aa868 in term 1.
I20260810 07:29:15.775089 10725 leader_election.cc:302] T e26e10ec787643acb8185fd841ccba0e P c4dd8966870f4b1a98474f0f6a6aa868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b276a048c139457286627db8cbb12a36, c4dd8966870f4b1a98474f0f6a6aa868; no voters:
I20260810 07:29:15.775344 11384 raft_consensus.cc:2781] T e26e10ec787643acb8185fd841ccba0e P c4dd8966870f4b1a98474f0f6a6aa868 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:15.775583 11384 raft_consensus.cc:686] T e26e10ec787643acb8185fd841ccba0e P c4dd8966870f4b1a98474f0f6a6aa868 [term 1 LEADER]: Becoming Leader. State: Replica: c4dd8966870f4b1a98474f0f6a6aa868, State: Running, Role: LEADER
I20260810 07:29:15.775908 11384 consensus_queue.cc:237] T e26e10ec787643acb8185fd841ccba0e P c4dd8966870f4b1a98474f0f6a6aa868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4dd8966870f4b1a98474f0f6a6aa868" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39267 } } peers { permanent_uuid: "22cd1f41716a47c4ab829b573c28f658" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34785 } } peers { permanent_uuid: "b276a048c139457286627db8cbb12a36" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 45109 } }
I20260810 07:29:15.779270 10590 catalog_manager.cc:5495] T e26e10ec787643acb8185fd841ccba0e P c4dd8966870f4b1a98474f0f6a6aa868 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4dd8966870f4b1a98474f0f6a6aa868 (127.12.157.193). New cstate: current_term: 1 leader_uuid: "c4dd8966870f4b1a98474f0f6a6aa868" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4dd8966870f4b1a98474f0f6a6aa868" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 39267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22cd1f41716a47c4ab829b573c28f658" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b276a048c139457286627db8cbb12a36" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 45109 } health_report { overall_health: UNKNOWN } } }
I20260810 07:29:15.807381 11364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:15.803719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42106 (local address 127.12.157.193:39267)
0810 07:29:15.803861 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:15.803865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:15.803879 (+ 14us) server_negotiation.cc:407] Connection header received
0810 07:29:15.803971 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:15.803973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:15.804030 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:15.804119 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:29:15.804359 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.804879 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.805844 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:15.806066 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:15.806152 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:15.806461 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:29:15.806885 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:29:15.807044 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:15.807164 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 07:29:15.807220 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 07:29:15.813462 11276 raft_consensus.cc:1261] T e26e10ec787643acb8185fd841ccba0e P 22cd1f41716a47c4ab829b573c28f658 [term 1 FOLLOWER]: Refusing update from remote peer c4dd8966870f4b1a98474f0f6a6aa868: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:29:15.813460 11044 raft_consensus.cc:1261] T e26e10ec787643acb8185fd841ccba0e P b276a048c139457286627db8cbb12a36 [term 1 FOLLOWER]: Refusing update from remote peer c4dd8966870f4b1a98474f0f6a6aa868: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:29:15.814250 11389 consensus_queue.cc:1035] T e26e10ec787643acb8185fd841ccba0e P c4dd8966870f4b1a98474f0f6a6aa868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22cd1f41716a47c4ab829b573c28f658" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:29:15.814438 11384 consensus_queue.cc:1035] T e26e10ec787643acb8185fd841ccba0e P c4dd8966870f4b1a98474f0f6a6aa868 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b276a048c139457286627db8cbb12a36" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 45109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:29:15.821748 11400 mvcc.cc:205] Tried to move back new op lower bound from 7316877131004186624 to 7316877130861092864. Current Snapshot: MvccSnapshot[applied={T|T < 7316877131004186624}]
I20260810 07:29:15.823191 11398 mvcc.cc:205] Tried to move back new op lower bound from 7316877131004186624 to 7316877130861092864. Current Snapshot: MvccSnapshot[applied={T|T < 7316877131004186624}]
I20260810 07:29:15.831216 10587 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:29:15.831382 10587 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:29:15.834100 10587 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 74e68dbebe7d41ffbae31e3e79c9abf5: Sending DeleteTablet for 3 replicas of tablet e26e10ec787643acb8185fd841ccba0e
I20260810 07:29:15.835044 10786 tablet_service.cc:1514] Processing DeleteTablet for tablet e26e10ec787643acb8185fd841ccba0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:29:15 UTC) from {username='slave'} at 127.0.0.1:42100
I20260810 07:29:15.835054 11004 tablet_service.cc:1514] Processing DeleteTablet for tablet e26e10ec787643acb8185fd841ccba0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:29:15 UTC) from {username='slave'} at 127.0.0.1:49942
I20260810 07:29:15.835057 11240 tablet_service.cc:1514] Processing DeleteTablet for tablet e26e10ec787643acb8185fd841ccba0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:29:15 UTC) from {username='slave'} at 127.0.0.1:39264
I20260810 07:29:15.835893 11417 tablet_replica.cc:330] T e26e10ec787643acb8185fd841ccba0e P 22cd1f41716a47c4ab829b573c28f658: stopping tablet replica
I20260810 07:29:15.836292 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 10692
I20260810 07:29:15.836292 11417 raft_consensus.cc:2229] T e26e10ec787643acb8185fd841ccba0e P 22cd1f41716a47c4ab829b573c28f658 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:29:15.836649 11417 raft_consensus.cc:2258] T e26e10ec787643acb8185fd841ccba0e P 22cd1f41716a47c4ab829b573c28f658 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:29:15.837234 11416 tablet_replica.cc:330] T e26e10ec787643acb8185fd841ccba0e P b276a048c139457286627db8cbb12a36: stopping tablet replica
I20260810 07:29:15.838058 11416 raft_consensus.cc:2229] T e26e10ec787643acb8185fd841ccba0e P b276a048c139457286627db8cbb12a36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:29:15.838837 11416 raft_consensus.cc:2258] T e26e10ec787643acb8185fd841ccba0e P b276a048c139457286627db8cbb12a36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:29:15.838845 11417 ts_tablet_manager.cc:1905] T e26e10ec787643acb8185fd841ccba0e P 22cd1f41716a47c4ab829b573c28f658: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:29:15.843001 11417 ts_tablet_manager.cc:1918] T e26e10ec787643acb8185fd841ccba0e P 22cd1f41716a47c4ab829b573c28f658: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:29:15.843817 11417 log.cc:1196] T e26e10ec787643acb8185fd841ccba0e P 22cd1f41716a47c4ab829b573c28f658: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-2/wal/wals/e26e10ec787643acb8185fd841ccba0e
I20260810 07:29:15.844316 11417 ts_tablet_manager.cc:1939] T e26e10ec787643acb8185fd841ccba0e P 22cd1f41716a47c4ab829b573c28f658: Deleting consensus metadata
I20260810 07:29:15.846519 11416 ts_tablet_manager.cc:1905] T e26e10ec787643acb8185fd841ccba0e P b276a048c139457286627db8cbb12a36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:29:15.846577 10563 catalog_manager.cc:4841] TS 22cd1f41716a47c4ab829b573c28f658 (127.12.157.195:34785): tablet e26e10ec787643acb8185fd841ccba0e (table test-table [id=1773ac2cbd3f467d86586319a821f115]) successfully deleted
I20260810 07:29:15.849069 11416 ts_tablet_manager.cc:1918] T e26e10ec787643acb8185fd841ccba0e P b276a048c139457286627db8cbb12a36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:29:15.849198 11416 log.cc:1196] T e26e10ec787643acb8185fd841ccba0e P b276a048c139457286627db8cbb12a36: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786346694975122-12919-0/minicluster-data/ts-1/wal/wals/e26e10ec787643acb8185fd841ccba0e
I20260810 07:29:15.849509 11416 ts_tablet_manager.cc:1939] T e26e10ec787643acb8185fd841ccba0e P b276a048c139457286627db8cbb12a36: Deleting consensus metadata
I20260810 07:29:15.850698 10564 catalog_manager.cc:4841] TS b276a048c139457286627db8cbb12a36 (127.12.157.194:45109): tablet e26e10ec787643acb8185fd841ccba0e (table test-table [id=1773ac2cbd3f467d86586319a821f115]) successfully deleted
W20260810 07:29:15.852754 10563 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.157.193:39267 (error 108)
I20260810 07:29:15.853611 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 10900
I20260810 07:29:15.853870 11365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:15.853558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:39267 (local address 127.0.0.1:42118)
0810 07:29:15.853696 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:15.853773 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.193:39267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 07:29:15.857697 10563 catalog_manager.cc:4584] TS c4dd8966870f4b1a98474f0f6a6aa868 (127.12.157.193:39267): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e26e10ec787643acb8185fd841ccba0e: Network error: Client connection negotiation failed: client connection to 127.12.157.193:39267: connect: Connection refused (error 111)
I20260810 07:29:15.874413 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 11134
I20260810 07:29:15.886245 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 10537
2026-08-10T07:29:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1166 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:29:15 dist-test-slave-ffrd krb5kdc[11426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:29:15 dist-test-slave-ffrd krb5kdc[11426](info): set up 2 sockets
Aug 10 07:29:15 dist-test-slave-ffrd krb5kdc[11426](info): commencing operation
krb5kdc: starting...
W20260810 07:29:17.183442 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.269s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:29:17 dist-test-slave-ffrd krb5kdc[11426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:29:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:29:17.319638 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:45855
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:44371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:45855
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:17.434779 11446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:17.435122 11446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:17.435201 11446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:17.439248 11446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:17.439348 11446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:17.439388 11446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:17.439450 11446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:17.439487 11446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:17.444741 11446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:44371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:45855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:45855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:17.445868 11446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:17.446933 11446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:17.455711 11458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:17.455746 11457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:17.455744 11462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:17.455864 11446 server_base.cc:1034] running on GCE node
I20260810 07:29:17.456619 11446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:17.457804 11446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:17.458967 11446 hybrid_clock.cc:648] HybridClock initialized: now 1786346957458948 us; error 31 us; skew 500 ppm
Aug 10 07:29:17 dist-test-slave-ffrd krb5kdc[11426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346957, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:17.461680 11446 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:29:17.462554 11446 webserver.cc:460] Webserver started at http://127.12.157.254:44699/ using document root <none> and password file <none>
I20260810 07:29:17.463084 11446 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:17.463142 11446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:17.463308 11446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:17.465394 11446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "894592c183754455bd16d9ce898b00ab"
format_stamp: "Formatted at 2026-08-10 07:29:17 on dist-test-slave-ffrd"
server_key: "1913e919ca0bd2abeed3b2c80b2c5609"
server_key_iv: "d95080cb6327e6c92f5347c9f098820a"
server_key_version: "encryptionkey@0"
I20260810 07:29:17.465914 11446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "894592c183754455bd16d9ce898b00ab"
format_stamp: "Formatted at 2026-08-10 07:29:17 on dist-test-slave-ffrd"
server_key: "1913e919ca0bd2abeed3b2c80b2c5609"
server_key_iv: "d95080cb6327e6c92f5347c9f098820a"
server_key_version: "encryptionkey@0"
I20260810 07:29:17.469666 11446 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 07:29:17.472137 11474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:17.473578 11446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:29:17.473739 11446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "894592c183754455bd16d9ce898b00ab"
format_stamp: "Formatted at 2026-08-10 07:29:17 on dist-test-slave-ffrd"
server_key: "1913e919ca0bd2abeed3b2c80b2c5609"
server_key_iv: "d95080cb6327e6c92f5347c9f098820a"
server_key_version: "encryptionkey@0"
I20260810 07:29:17.473863 11446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:17.495812 11446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:17.499248 11446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:17.499467 11446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:17.507055 11446 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:45855
I20260810 07:29:17.507103 11564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:45855 every 8 connection(s)
I20260810 07:29:17.508047 11446 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:17.510985 11565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:17.516574 11565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 894592c183754455bd16d9ce898b00ab: Bootstrap starting.
I20260810 07:29:17.517517 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 11446
I20260810 07:29:17.517649 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:29:17.517925 12919 external_mini_cluster.cc:1428] Setting key 3339c333e021f881c4f998e221067c23
I20260810 07:29:17.519467 11565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 894592c183754455bd16d9ce898b00ab: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:17.520404 11565 log.cc:824] T 00000000000000000000000000000000 P 894592c183754455bd16d9ce898b00ab: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:17.522738 11565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 894592c183754455bd16d9ce898b00ab: No bootstrap required, opened a new log
I20260810 07:29:17.526854 11565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 894592c183754455bd16d9ce898b00ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894592c183754455bd16d9ce898b00ab" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 45855 } }
I20260810 07:29:17.527078 11565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 894592c183754455bd16d9ce898b00ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:17.527114 11565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 894592c183754455bd16d9ce898b00ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 894592c183754455bd16d9ce898b00ab, State: Initialized, Role: FOLLOWER
I20260810 07:29:17.527631 11565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 894592c183754455bd16d9ce898b00ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894592c183754455bd16d9ce898b00ab" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 45855 } }
Aug 10 07:29:17 dist-test-slave-ffrd krb5kdc[11426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:29:17.527805 11565 raft_consensus.cc:388] T 00000000000000000000000000000000 P 894592c183754455bd16d9ce898b00ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:29:17.527889 11565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 894592c183754455bd16d9ce898b00ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:29:17.528018 11565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 894592c183754455bd16d9ce898b00ab [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:17.529323 11565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 894592c183754455bd16d9ce898b00ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894592c183754455bd16d9ce898b00ab" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 45855 } }
I20260810 07:29:17.529728 11565 leader_election.cc:302] T 00000000000000000000000000000000 P 894592c183754455bd16d9ce898b00ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 894592c183754455bd16d9ce898b00ab; no voters:
I20260810 07:29:17.530160 11565 leader_election.cc:288] T 00000000000000000000000000000000 P 894592c183754455bd16d9ce898b00ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:29:17.530233 11572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 894592c183754455bd16d9ce898b00ab [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:17.530468 11572 raft_consensus.cc:686] T 00000000000000000000000000000000 P 894592c183754455bd16d9ce898b00ab [term 1 LEADER]: Becoming Leader. State: Replica: 894592c183754455bd16d9ce898b00ab, State: Running, Role: LEADER
I20260810 07:29:17.530831 11572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 894592c183754455bd16d9ce898b00ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894592c183754455bd16d9ce898b00ab" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 45855 } }
I20260810 07:29:17.531544 11565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 894592c183754455bd16d9ce898b00ab [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:17.532882 11573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 894592c183754455bd16d9ce898b00ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "894592c183754455bd16d9ce898b00ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894592c183754455bd16d9ce898b00ab" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 45855 } } }
I20260810 07:29:17.533054 11573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 894592c183754455bd16d9ce898b00ab [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:17.534119 11574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 894592c183754455bd16d9ce898b00ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 894592c183754455bd16d9ce898b00ab. Latest consensus state: current_term: 1 leader_uuid: "894592c183754455bd16d9ce898b00ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "894592c183754455bd16d9ce898b00ab" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 45855 } } }
I20260810 07:29:17.534236 11574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 894592c183754455bd16d9ce898b00ab [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:17.535014 11568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:17.519109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39564 (local address 127.12.157.254:45855)
0810 07:29:17.519756 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:17.519777 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:17.519825 (+ 48us) server_negotiation.cc:407] Connection header received
0810 07:29:17.520701 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:17.520727 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:17.521158 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:17.521524 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:17.522264 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:17.523731 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:17.524664 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:17.525456 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:17.525659 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:17.528440 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:17.528447 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:17.528464 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:17.528520 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:17.530683 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:17.531204 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:17.531210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:17.531215 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:17.531365 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:17.531698 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:17.531701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:17.531703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:17.532282 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:17.532503 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:17.533853 (+ 1350us) server_negotiation.cc:299] Negotiation successful
0810 07:29:17.534207 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":263,"threads_started":1}
W20260810 07:29:17.536756 11597 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 894592c183754455bd16d9ce898b00ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:29:17.536834 11597 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:29:17.536900 11588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:17.537747 11588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:17.543448 11588 catalog_manager.cc:1347] Generated new cluster ID: 68927dc03faa448c981e9c19a26be87b
I20260810 07:29:17.543538 11588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:17.569270 11588 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:17.570325 11588 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:17.576025 11588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 894592c183754455bd16d9ce898b00ab: Generated new TSK 0
I20260810 07:29:17.576799 11588 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.157.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
Entry for principal kudu/127.12.157.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab.
I20260810 07:29:17.611937 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:0
--local_ip_for_outbound_sockets=127.12.157.193
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:45855
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:44371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:17.730609 11609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:17.730901 11609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:17.730974 11609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:17.735317 11609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:17.735404 11609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:17.735502 11609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:29:17.740839 11609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:44371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.193.keytab
--principal=kudu/127.12.157.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:45855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:17.741947 11609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:17.742961 11609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:17.753135 11616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:17.753130 11620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:17.753149 11609 server_base.cc:1034] running on GCE node
W20260810 07:29:17.753130 11618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:17.753834 11609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:17.754597 11609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:17.755841 11609 hybrid_clock.cc:648] HybridClock initialized: now 1786346957755826 us; error 71 us; skew 500 ppm
Aug 10 07:29:17 dist-test-slave-ffrd krb5kdc[11426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346957, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:17.758914 11609 init.cc:377] Logged in from keytab as kudu/127.12.157.193@KRBTEST.COM (short username kudu)
I20260810 07:29:17.759953 11609 webserver.cc:460] Webserver started at http://127.12.157.193:36901/ using document root <none> and password file <none>
I20260810 07:29:17.760540 11609 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:17.760633 11609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:17.760883 11609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:17.763075 11609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-0/data/instance:
uuid: "05fc57b1c97349e9916c660e5760b5dc"
format_stamp: "Formatted at 2026-08-10 07:29:17 on dist-test-slave-ffrd"
server_key: "ffb51f5bc2ee90298af89951c1cbd106"
server_key_iv: "419e0c288c1bff7cadd0651a72317e05"
server_key_version: "encryptionkey@0"
I20260810 07:29:17.763666 11609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance:
uuid: "05fc57b1c97349e9916c660e5760b5dc"
format_stamp: "Formatted at 2026-08-10 07:29:17 on dist-test-slave-ffrd"
server_key: "ffb51f5bc2ee90298af89951c1cbd106"
server_key_iv: "419e0c288c1bff7cadd0651a72317e05"
server_key_version: "encryptionkey@0"
I20260810 07:29:17.767625 11609 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 07:29:17.770211 11627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:17.771543 11609 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 07:29:17.771708 11609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "05fc57b1c97349e9916c660e5760b5dc"
format_stamp: "Formatted at 2026-08-10 07:29:17 on dist-test-slave-ffrd"
server_key: "ffb51f5bc2ee90298af89951c1cbd106"
server_key_iv: "419e0c288c1bff7cadd0651a72317e05"
server_key_version: "encryptionkey@0"
I20260810 07:29:17.771858 11609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:17.786111 11609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:17.789654 11609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:17.789873 11609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:17.790547 11609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:29:17.791441 11609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:29:17.791512 11609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:17.791584 11609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:29:17.791633 11609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:17.801581 11609 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:38889
I20260810 07:29:17.801620 11809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:38889 every 8 connection(s)
I20260810 07:29:17.802518 11609 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:29:17.810735 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 11609
I20260810 07:29:17.810921 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance
I20260810 07:29:17.811244 12919 external_mini_cluster.cc:1428] Setting key d59f3571e8c4ba03a0d2b37bebe1fb2c
Aug 10 07:29:17 dist-test-slave-ffrd krb5kdc[11426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346957, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.193@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:29:17.817272 11568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:17.804361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:34107 (local address 127.12.157.254:45855)
0810 07:29:17.804507 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:17.804511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:17.805371 (+ 860us) server_negotiation.cc:407] Connection header received
0810 07:29:17.806740 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:17.806743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:17.806804 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:17.806914 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:17.808064 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:17.808604 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:17.809818 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:17.810056 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:17.810170 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:17.813077 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:17.813081 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:17.813083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:17.813106 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:17.814753 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:17.815447 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:17.815453 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:17.815456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:17.815562 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:17.815911 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:17.815917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:17.815920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:17.816186 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:17.816296 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:17.816828 (+ 532us) server_negotiation.cc:299] Negotiation successful
0810 07:29:17.817023 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 07:29:17.818005 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:17.804704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:45855 (local address 127.12.157.193:34107)
0810 07:29:17.805215 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:17.805257 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:17.806514 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:17.807083 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:17.807090 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:17.807447 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:17.807918 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:17.807932 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:17.808730 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:17.808739 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:17.809680 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:17.809690 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:17.810213 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:17.810217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:17.810388 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:17.811076 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:17.811092 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:17.812877 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:17.814891 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:17.814901 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:17.814921 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:17.815325 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:17.815658 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:17.815660 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:17.815661 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:17.815804 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:17.816303 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:17.816308 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:17.816558 (+ 250us) client_negotiation.cc:769] Sending connection context
0810 07:29:17.816716 (+ 158us) client_negotiation.cc:240] Negotiation successful
0810 07:29:17.817073 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":128,"threads_started":1}
I20260810 07:29:17.819479 11811 heartbeater.cc:347] Connected to a master server at 127.12.157.254:45855
I20260810 07:29:17.819747 11811 heartbeater.cc:464] Registering TS with master...
I20260810 07:29:17.820297 11811 heartbeater.cc:511] Master 127.12.157.254:45855 requested a full tablet report, sending...
I20260810 07:29:17.822268 11505 ts_manager.cc:194] Registered new tserver with Master: 05fc57b1c97349e9916c660e5760b5dc (127.12.157.193:38889)
WARNING: no policy specified for kudu/127.12.157.194@KRBTEST.COM; defaulting to no policy
I20260810 07:29:17.823904 11505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.193@KRBTEST.COM'} at 127.12.157.193:34107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
Entry for principal kudu/127.12.157.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab.
I20260810 07:29:17.843926 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:0
--local_ip_for_outbound_sockets=127.12.157.194
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:45855
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:44371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:17.963335 11818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:17.963596 11818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:17.963640 11818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:17.967573 11818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:17.967635 11818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:17.967694 11818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:29:17.973021 11818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:44371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.194.keytab
--principal=kudu/127.12.157.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:45855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:17.974066 11818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:17.975059 11818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:17.985448 11825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:17.985433 11824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:17.985440 11828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:17.985555 11818 server_base.cc:1034] running on GCE node
I20260810 07:29:17.986069 11818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:17.986752 11818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:17.987959 11818 hybrid_clock.cc:648] HybridClock initialized: now 1786346957987948 us; error 26 us; skew 500 ppm
Aug 10 07:29:17 dist-test-slave-ffrd krb5kdc[11426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346957, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:17.996704 11818 init.cc:377] Logged in from keytab as kudu/127.12.157.194@KRBTEST.COM (short username kudu)
I20260810 07:29:17.997750 11818 webserver.cc:460] Webserver started at http://127.12.157.194:37509/ using document root <none> and password file <none>
I20260810 07:29:17.998353 11818 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:17.998417 11818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:17.998641 11818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:18.000720 11818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-1/data/instance:
uuid: "f7fdcea914c0484ebe99e5bf0e4e6f05"
format_stamp: "Formatted at 2026-08-10 07:29:17 on dist-test-slave-ffrd"
server_key: "e3ec77625204ccc749973aede87aa4c3"
server_key_iv: "b13c9993e9ed39306cbe739529cfec08"
server_key_version: "encryptionkey@0"
I20260810 07:29:18.001367 11818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance:
uuid: "f7fdcea914c0484ebe99e5bf0e4e6f05"
format_stamp: "Formatted at 2026-08-10 07:29:17 on dist-test-slave-ffrd"
server_key: "e3ec77625204ccc749973aede87aa4c3"
server_key_iv: "b13c9993e9ed39306cbe739529cfec08"
server_key_version: "encryptionkey@0"
I20260810 07:29:18.005250 11818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 07:29:18.007714 11839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:18.009011 11818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:29:18.009141 11818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "f7fdcea914c0484ebe99e5bf0e4e6f05"
format_stamp: "Formatted at 2026-08-10 07:29:17 on dist-test-slave-ffrd"
server_key: "e3ec77625204ccc749973aede87aa4c3"
server_key_iv: "b13c9993e9ed39306cbe739529cfec08"
server_key_version: "encryptionkey@0"
I20260810 07:29:18.009258 11818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:18.024603 11818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:18.027977 11818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:18.028188 11818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:18.028831 11818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:29:18.029731 11818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:29:18.029789 11818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:18.029860 11818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:29:18.029893 11818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:18.039489 11818 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:45889
I20260810 07:29:18.039517 12026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:45889 every 8 connection(s)
I20260810 07:29:18.040392 11818 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:29:18.042755 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 11818
I20260810 07:29:18.042899 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance
I20260810 07:29:18.043228 12919 external_mini_cluster.cc:1428] Setting key c9c65d48782ee6ed63bd10c7c2508ee9
Aug 10 07:29:18 dist-test-slave-ffrd krb5kdc[11426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346957, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.194@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.157.195@KRBTEST.COM; defaulting to no policy
I20260810 07:29:18.056211 11568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:18.042299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:52161 (local address 127.12.157.254:45855)
0810 07:29:18.042464 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:18.042468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:18.043453 (+ 985us) server_negotiation.cc:407] Connection header received
0810 07:29:18.044896 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:18.044899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:18.044985 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:18.045067 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:18.046721 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.047536 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.048739 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.049065 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.049180 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:18.052215 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:18.052221 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:18.052229 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:18.052255 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:18.053958 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:18.054517 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:18.054521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:18.054523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:18.054607 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:18.054963 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:18.054967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:18.054969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:18.055195 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:18.055301 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:18.055909 (+ 608us) server_negotiation.cc:299] Negotiation successful
0810 07:29:18.056037 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 07:29:18.057190 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:18.042675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:45855 (local address 127.12.157.194:52161)
0810 07:29:18.043262 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:18.043325 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:18.044660 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:18.045275 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:18.045286 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:18.045821 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:18.046540 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:18.046563 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.047705 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.047708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:18.048615 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:18.048624 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.049242 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.049246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:18.049429 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:18.050095 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:18.050112 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:18.052022 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:18.054093 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:18.054100 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:18.054117 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:18.054405 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:18.054710 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:18.054713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:18.054715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:18.054862 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:18.055325 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:18.055332 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:18.055598 (+ 266us) client_negotiation.cc:769] Sending connection context
0810 07:29:18.055824 (+ 226us) client_negotiation.cc:240] Negotiation successful
0810 07:29:18.056182 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":198,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.195@KRBTEST.COM" created.
I20260810 07:29:18.058422 12027 heartbeater.cc:347] Connected to a master server at 127.12.157.254:45855
I20260810 07:29:18.058698 12027 heartbeater.cc:464] Registering TS with master...
I20260810 07:29:18.059316 12027 heartbeater.cc:511] Master 127.12.157.254:45855 requested a full tablet report, sending...
I20260810 07:29:18.060565 11505 ts_manager.cc:194] Registered new tserver with Master: f7fdcea914c0484ebe99e5bf0e4e6f05 (127.12.157.194:45889)
I20260810 07:29:18.061239 11505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.194@KRBTEST.COM'} at 127.12.157.194:52161
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
Entry for principal kudu/127.12.157.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab.
I20260810 07:29:18.074337 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:0
--local_ip_for_outbound_sockets=127.12.157.195
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:45855
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:44371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:18.209527 12036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:18.209987 12036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:18.210062 12036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:18.217206 12036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:18.217298 12036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:18.217392 12036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:29:18.223373 12036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:44371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.195.keytab
--principal=kudu/127.12.157.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:45855
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:18.224545 12036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:18.225646 12036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:18.235890 12044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:18.235957 12049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:18.235977 12045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:18.236135 12036 server_base.cc:1034] running on GCE node
I20260810 07:29:18.236577 12036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:18.237298 12036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:18.238471 12036 hybrid_clock.cc:648] HybridClock initialized: now 1786346958238430 us; error 51 us; skew 500 ppm
Aug 10 07:29:18 dist-test-slave-ffrd krb5kdc[11426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346958, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:18.241427 12036 init.cc:377] Logged in from keytab as kudu/127.12.157.195@KRBTEST.COM (short username kudu)
I20260810 07:29:18.242378 12036 webserver.cc:460] Webserver started at http://127.12.157.195:38031/ using document root <none> and password file <none>
I20260810 07:29:18.242998 12036 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:18.243116 12036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:18.243352 12036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:18.245600 12036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-2/data/instance:
uuid: "b43516e2cada4e9da741b39ff9232154"
format_stamp: "Formatted at 2026-08-10 07:29:18 on dist-test-slave-ffrd"
server_key: "c03dbdfb8f05e9fe873c42d4501eec32"
server_key_iv: "b42563d46804592e5dadaf9bef3a8fca"
server_key_version: "encryptionkey@0"
I20260810 07:29:18.246203 12036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance:
uuid: "b43516e2cada4e9da741b39ff9232154"
format_stamp: "Formatted at 2026-08-10 07:29:18 on dist-test-slave-ffrd"
server_key: "c03dbdfb8f05e9fe873c42d4501eec32"
server_key_iv: "b42563d46804592e5dadaf9bef3a8fca"
server_key_version: "encryptionkey@0"
I20260810 07:29:18.250187 12036 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 07:29:18.253068 12064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:18.254451 12036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:29:18.254607 12036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "b43516e2cada4e9da741b39ff9232154"
format_stamp: "Formatted at 2026-08-10 07:29:18 on dist-test-slave-ffrd"
server_key: "c03dbdfb8f05e9fe873c42d4501eec32"
server_key_iv: "b42563d46804592e5dadaf9bef3a8fca"
server_key_version: "encryptionkey@0"
I20260810 07:29:18.254737 12036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:18.277261 12036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:18.280649 12036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:18.280892 12036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:18.281566 12036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:29:18.282620 12036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:29:18.282711 12036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:18.282790 12036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:29:18.282836 12036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:18.292294 12036 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:34563
I20260810 07:29:18.292340 12235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:34563 every 8 connection(s)
I20260810 07:29:18.293262 12036 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:29:18.296123 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 12036
I20260810 07:29:18.296273 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance
I20260810 07:29:18.296636 12919 external_mini_cluster.cc:1428] Setting key ea1797d1a52fc3d4ad1668fe7a34c618
Aug 10 07:29:18 dist-test-slave-ffrd krb5kdc[11426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346958, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.195@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:29:18.308161 11568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:18.295147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:52533 (local address 127.12.157.254:45855)
0810 07:29:18.295293 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:18.295296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:18.296261 (+ 965us) server_negotiation.cc:407] Connection header received
0810 07:29:18.298087 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:18.298091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:18.298164 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:18.298255 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:18.299609 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.300160 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.301397 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.301627 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.301722 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:18.304484 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:18.304488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:18.304490 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:18.304517 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:18.306128 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:18.306652 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:18.306655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:18.306656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:18.306744 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:18.307135 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:18.307138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:18.307139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:18.307331 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:18.307422 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:18.307926 (+ 504us) server_negotiation.cc:299] Negotiation successful
0810 07:29:18.308034 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 07:29:18.309084 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:18.295527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:45855 (local address 127.12.157.195:52533)
0810 07:29:18.296080 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:18.296146 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:18.297821 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:18.298461 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:18.298468 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:18.298822 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:18.299470 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:18.299485 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.300273 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.300275 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:18.301266 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:18.301277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.301794 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.301798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:18.301967 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:18.302564 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:18.302581 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:18.304286 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:18.306239 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:18.306245 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:18.306257 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:18.306537 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:18.306830 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:18.306832 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:18.306833 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:18.307022 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:18.307432 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:18.307437 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:18.307686 (+ 249us) client_negotiation.cc:769] Sending connection context
0810 07:29:18.307867 (+ 181us) client_negotiation.cc:240] Negotiation successful
0810 07:29:18.308149 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":125,"threads_started":1}
I20260810 07:29:18.310369 12236 heartbeater.cc:347] Connected to a master server at 127.12.157.254:45855
I20260810 07:29:18.310667 12236 heartbeater.cc:464] Registering TS with master...
I20260810 07:29:18.311228 12236 heartbeater.cc:511] Master 127.12.157.254:45855 requested a full tablet report, sending...
I20260810 07:29:18.312304 11505 ts_manager.cc:194] Registered new tserver with Master: b43516e2cada4e9da741b39ff9232154 (127.12.157.195:34563)
I20260810 07:29:18.313088 11505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.157.195@KRBTEST.COM'} at 127.12.157.195:52533
I20260810 07:29:18.322765 12919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:29:18.332202 11568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:18.324894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39574 (local address 127.12.157.254:45855)
0810 07:29:18.325074 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:18.325077 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:18.325224 (+ 147us) server_negotiation.cc:407] Connection header received
0810 07:29:18.325334 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:18.325336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:18.325377 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:18.325444 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:18.326186 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.326694 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.327564 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.327772 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.327857 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:18.329280 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:18.329284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:18.329286 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:18.329310 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:18.330602 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:18.331001 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:18.331003 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:18.331004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:18.331076 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:18.331346 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:18.331350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:18.331352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:18.331569 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:18.331675 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:18.331942 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 07:29:18.332033 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260810 07:29:18.335201 11505 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:29:18.338271 11505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:29:18.353468 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:18.346932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:34563 (local address 127.0.0.1:59676)
0810 07:29:18.347530 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:18.347585 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:18.348116 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:18.349158 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:18.349161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:18.349189 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:18.349354 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:18.349360 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.350602 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.350605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:18.351771 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:18.351781 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.352823 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.352826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:18.352926 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:18.353037 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 07:29:18.353177 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 07:29:18.353305 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":103,"spinlock_wait_cycles":5376,"thread_start_us":79,"threads_started":1}
I20260810 07:29:18.353768 12257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:18.346991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:38889 (local address 127.0.0.1:51936)
0810 07:29:18.347763 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:18.347778 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:18.347916 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:18.349000 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:18.349006 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:18.349029 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:18.349169 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:18.349177 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.350688 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.350693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:18.352230 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:18.352242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.353395 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.353399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:18.353494 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:18.353519 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 07:29:18.353565 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 07:29:18.353632 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":166,"thread_start_us":106,"threads_started":1}
I20260810 07:29:18.354049 12260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:18.347096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59676 (local address 127.12.157.195:34563)
0810 07:29:18.348595 (+ 1499us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:18.348604 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:18.348622 (+ 18us) server_negotiation.cc:407] Connection header received
0810 07:29:18.348684 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:18.348688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:18.348824 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:18.349003 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:18.349473 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.350452 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.351922 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.352677 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.352821 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:18.352996 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:18.353217 (+ 221us) server_negotiation.cc:299] Negotiation successful
0810 07:29:18.353318 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1411,"thread_start_us":137,"threads_started":1}
I20260810 07:29:18.354655 12261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:18.347697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:45889 (local address 127.0.0.1:40058)
0810 07:29:18.348151 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:18.348163 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:18.348240 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:18.349145 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:18.349147 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:18.349173 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:18.349350 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:18.349357 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.350739 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.350742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:18.352230 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:18.352242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.354249 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.354253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:18.354353 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:18.354380 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 07:29:18.354457 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 07:29:18.354525 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":168,"spinlock_wait_cycles":5632,"thread_start_us":100,"threads_started":1}
I20260810 07:29:18.354755 12265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:18.348135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40058 (local address 127.12.157.194:45889)
0810 07:29:18.348501 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:18.348513 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:18.348533 (+ 20us) server_negotiation.cc:407] Connection header received
0810 07:29:18.348597 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:18.348601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:18.348798 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:18.348980 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:18.349492 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.350607 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.353070 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.353922 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.354045 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:18.354186 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:18.354512 (+ 326us) server_negotiation.cc:299] Negotiation successful
0810 07:29:18.354609 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":131,"threads_started":1}
I20260810 07:29:18.355020 12262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:18.347743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51936 (local address 127.12.157.193:38889)
0810 07:29:18.348479 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:18.348487 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:18.348512 (+ 25us) server_negotiation.cc:407] Connection header received
0810 07:29:18.348595 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:18.348600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:18.348799 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:18.348980 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:18.349301 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.350542 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.352371 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.353240 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.354407 (+ 1167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:18.354544 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:18.354594 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 07:29:18.354684 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":146,"threads_started":1}
I20260810 07:29:18.356992 12142 tablet_service.cc:1467] Processing CreateTablet for tablet 10310b7d38534687b37ead375be7b64c (DEFAULT_TABLE table=test-table [id=c2fa34efd3104878bf8ab1ca038fea4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:29:18.357576 11925 tablet_service.cc:1467] Processing CreateTablet for tablet 10310b7d38534687b37ead375be7b64c (DEFAULT_TABLE table=test-table [id=c2fa34efd3104878bf8ab1ca038fea4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:29:18.357574 11708 tablet_service.cc:1467] Processing CreateTablet for tablet 10310b7d38534687b37ead375be7b64c (DEFAULT_TABLE table=test-table [id=c2fa34efd3104878bf8ab1ca038fea4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:29:18.358166 12142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10310b7d38534687b37ead375be7b64c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:18.358539 11925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10310b7d38534687b37ead375be7b64c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:18.358539 11708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10310b7d38534687b37ead375be7b64c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:18.364435 12271 tablet_bootstrap.cc:492] T 10310b7d38534687b37ead375be7b64c P b43516e2cada4e9da741b39ff9232154: Bootstrap starting.
I20260810 07:29:18.365986 12273 tablet_bootstrap.cc:492] T 10310b7d38534687b37ead375be7b64c P f7fdcea914c0484ebe99e5bf0e4e6f05: Bootstrap starting.
I20260810 07:29:18.366389 12272 tablet_bootstrap.cc:492] T 10310b7d38534687b37ead375be7b64c P 05fc57b1c97349e9916c660e5760b5dc: Bootstrap starting.
I20260810 07:29:18.366928 12271 tablet_bootstrap.cc:654] T 10310b7d38534687b37ead375be7b64c P b43516e2cada4e9da741b39ff9232154: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:18.367765 12273 tablet_bootstrap.cc:654] T 10310b7d38534687b37ead375be7b64c P f7fdcea914c0484ebe99e5bf0e4e6f05: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:18.368322 12271 log.cc:824] T 10310b7d38534687b37ead375be7b64c P b43516e2cada4e9da741b39ff9232154: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:18.368481 12272 tablet_bootstrap.cc:654] T 10310b7d38534687b37ead375be7b64c P 05fc57b1c97349e9916c660e5760b5dc: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:18.369151 12273 log.cc:824] T 10310b7d38534687b37ead375be7b64c P f7fdcea914c0484ebe99e5bf0e4e6f05: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:18.369288 12272 log.cc:824] T 10310b7d38534687b37ead375be7b64c P 05fc57b1c97349e9916c660e5760b5dc: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:18.370484 12271 tablet_bootstrap.cc:492] T 10310b7d38534687b37ead375be7b64c P b43516e2cada4e9da741b39ff9232154: No bootstrap required, opened a new log
I20260810 07:29:18.370699 12271 ts_tablet_manager.cc:1397] T 10310b7d38534687b37ead375be7b64c P b43516e2cada4e9da741b39ff9232154: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 07:29:18.371055 12272 tablet_bootstrap.cc:492] T 10310b7d38534687b37ead375be7b64c P 05fc57b1c97349e9916c660e5760b5dc: No bootstrap required, opened a new log
I20260810 07:29:18.371055 12273 tablet_bootstrap.cc:492] T 10310b7d38534687b37ead375be7b64c P f7fdcea914c0484ebe99e5bf0e4e6f05: No bootstrap required, opened a new log
I20260810 07:29:18.371236 12273 ts_tablet_manager.cc:1397] T 10310b7d38534687b37ead375be7b64c P f7fdcea914c0484ebe99e5bf0e4e6f05: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 07:29:18.371236 12272 ts_tablet_manager.cc:1397] T 10310b7d38534687b37ead375be7b64c P 05fc57b1c97349e9916c660e5760b5dc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 07:29:18.373711 12271 raft_consensus.cc:348] T 10310b7d38534687b37ead375be7b64c P b43516e2cada4e9da741b39ff9232154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b43516e2cada4e9da741b39ff9232154" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34563 } } peers { permanent_uuid: "05fc57b1c97349e9916c660e5760b5dc" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 38889 } } peers { permanent_uuid: "f7fdcea914c0484ebe99e5bf0e4e6f05" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 45889 } }
I20260810 07:29:18.374015 12271 raft_consensus.cc:374] T 10310b7d38534687b37ead375be7b64c P b43516e2cada4e9da741b39ff9232154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:18.374115 12271 raft_consensus.cc:729] T 10310b7d38534687b37ead375be7b64c P b43516e2cada4e9da741b39ff9232154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b43516e2cada4e9da741b39ff9232154, State: Initialized, Role: FOLLOWER
I20260810 07:29:18.374656 12271 consensus_queue.cc:260] T 10310b7d38534687b37ead375be7b64c P b43516e2cada4e9da741b39ff9232154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b43516e2cada4e9da741b39ff9232154" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34563 } } peers { permanent_uuid: "05fc57b1c97349e9916c660e5760b5dc" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 38889 } } peers { permanent_uuid: "f7fdcea914c0484ebe99e5bf0e4e6f05" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 45889 } }
I20260810 07:29:18.375306 12236 heartbeater.cc:503] Master 127.12.157.254:45855 was elected leader, sending a full tablet report...
I20260810 07:29:18.375676 12271 ts_tablet_manager.cc:1428] T 10310b7d38534687b37ead375be7b64c P b43516e2cada4e9da741b39ff9232154: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 07:29:18.376277 12273 raft_consensus.cc:348] T 10310b7d38534687b37ead375be7b64c P f7fdcea914c0484ebe99e5bf0e4e6f05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b43516e2cada4e9da741b39ff9232154" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34563 } } peers { permanent_uuid: "05fc57b1c97349e9916c660e5760b5dc" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 38889 } } peers { permanent_uuid: "f7fdcea914c0484ebe99e5bf0e4e6f05" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 45889 } }
I20260810 07:29:18.376579 12273 raft_consensus.cc:374] T 10310b7d38534687b37ead375be7b64c P f7fdcea914c0484ebe99e5bf0e4e6f05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:18.376643 12273 raft_consensus.cc:729] T 10310b7d38534687b37ead375be7b64c P f7fdcea914c0484ebe99e5bf0e4e6f05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7fdcea914c0484ebe99e5bf0e4e6f05, State: Initialized, Role: FOLLOWER
I20260810 07:29:18.376747 12272 raft_consensus.cc:348] T 10310b7d38534687b37ead375be7b64c P 05fc57b1c97349e9916c660e5760b5dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b43516e2cada4e9da741b39ff9232154" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34563 } } peers { permanent_uuid: "05fc57b1c97349e9916c660e5760b5dc" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 38889 } } peers { permanent_uuid: "f7fdcea914c0484ebe99e5bf0e4e6f05" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 45889 } }
I20260810 07:29:18.377130 12272 raft_consensus.cc:374] T 10310b7d38534687b37ead375be7b64c P 05fc57b1c97349e9916c660e5760b5dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:18.377115 12273 consensus_queue.cc:260] T 10310b7d38534687b37ead375be7b64c P f7fdcea914c0484ebe99e5bf0e4e6f05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b43516e2cada4e9da741b39ff9232154" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34563 } } peers { permanent_uuid: "05fc57b1c97349e9916c660e5760b5dc" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 38889 } } peers { permanent_uuid: "f7fdcea914c0484ebe99e5bf0e4e6f05" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 45889 } }
I20260810 07:29:18.377208 12272 raft_consensus.cc:729] T 10310b7d38534687b37ead375be7b64c P 05fc57b1c97349e9916c660e5760b5dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05fc57b1c97349e9916c660e5760b5dc, State: Initialized, Role: FOLLOWER
I20260810 07:29:18.377667 12272 consensus_queue.cc:260] T 10310b7d38534687b37ead375be7b64c P 05fc57b1c97349e9916c660e5760b5dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b43516e2cada4e9da741b39ff9232154" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34563 } } peers { permanent_uuid: "05fc57b1c97349e9916c660e5760b5dc" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 38889 } } peers { permanent_uuid: "f7fdcea914c0484ebe99e5bf0e4e6f05" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 45889 } }
I20260810 07:29:18.378008 12027 heartbeater.cc:503] Master 127.12.157.254:45855 was elected leader, sending a full tablet report...
I20260810 07:29:18.378090 12273 ts_tablet_manager.cc:1428] T 10310b7d38534687b37ead375be7b64c P f7fdcea914c0484ebe99e5bf0e4e6f05: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 07:29:18.378602 11811 heartbeater.cc:503] Master 127.12.157.254:45855 was elected leader, sending a full tablet report...
I20260810 07:29:18.378667 12272 ts_tablet_manager.cc:1428] T 10310b7d38534687b37ead375be7b64c P 05fc57b1c97349e9916c660e5760b5dc: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 07:29:18.476747 12283 raft_consensus.cc:482] T 10310b7d38534687b37ead375be7b64c P f7fdcea914c0484ebe99e5bf0e4e6f05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:29:18.477082 12283 raft_consensus.cc:504] T 10310b7d38534687b37ead375be7b64c P f7fdcea914c0484ebe99e5bf0e4e6f05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b43516e2cada4e9da741b39ff9232154" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34563 } } peers { permanent_uuid: "05fc57b1c97349e9916c660e5760b5dc" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 38889 } } peers { permanent_uuid: "f7fdcea914c0484ebe99e5bf0e4e6f05" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 45889 } }
I20260810 07:29:18.478430 12283 leader_election.cc:288] T 10310b7d38534687b37ead375be7b64c P f7fdcea914c0484ebe99e5bf0e4e6f05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b43516e2cada4e9da741b39ff9232154 (127.12.157.195:34563), 05fc57b1c97349e9916c660e5760b5dc (127.12.157.193:38889)
I20260810 07:29:18.482466 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:18.478767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:38889 (local address 127.12.157.194:40639)
0810 07:29:18.478994 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:18.479011 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:18.479115 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:18.479451 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:18.479454 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:18.479472 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:18.479598 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:18.479604 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.480349 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.480354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:18.481272 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:18.481282 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.481969 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.481973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:18.482136 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:18.482155 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 07:29:18.482238 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 07:29:18.482319 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":50}
I20260810 07:29:18.482467 12262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:18.479073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:40639 (local address 127.12.157.193:38889)
0810 07:29:18.479209 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:18.479214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:18.479235 (+ 21us) server_negotiation.cc:407] Connection header received
0810 07:29:18.479284 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:18.479287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:18.479327 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:18.479427 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:18.479720 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.480229 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.481408 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.481827 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.481971 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:18.482029 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:18.482268 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 07:29:18.482348 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 07:29:18.482663 12286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:18.478821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:34563 (local address 127.12.157.194:55283)
0810 07:29:18.479272 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:18.479292 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:18.479424 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:18.479698 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:18.479700 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:18.479717 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:18.479857 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:18.479863 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.480586 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.480590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:18.481393 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:18.481402 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.482050 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.482054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:18.482217 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:18.482235 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 07:29:18.482328 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 07:29:18.482414 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":25,"thread_start_us":174,"threads_started":1}
I20260810 07:29:18.482759 12260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:18.478910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:55283 (local address 127.12.157.195:34563)
0810 07:29:18.479086 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:18.479092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:18.479382 (+ 290us) server_negotiation.cc:407] Connection header received
0810 07:29:18.479543 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:18.479548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:18.479598 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:18.479684 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:18.479975 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.480469 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.481520 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.481922 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.482054 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:18.482119 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:18.482490 (+ 371us) server_negotiation.cc:299] Negotiation successful
0810 07:29:18.482635 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 07:29:18.483405 12175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10310b7d38534687b37ead375be7b64c" candidate_uuid: "f7fdcea914c0484ebe99e5bf0e4e6f05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b43516e2cada4e9da741b39ff9232154" is_pre_election: true
I20260810 07:29:18.483410 11734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10310b7d38534687b37ead375be7b64c" candidate_uuid: "f7fdcea914c0484ebe99e5bf0e4e6f05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05fc57b1c97349e9916c660e5760b5dc" is_pre_election: true
I20260810 07:29:18.483728 11734 raft_consensus.cc:2454] T 10310b7d38534687b37ead375be7b64c P 05fc57b1c97349e9916c660e5760b5dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7fdcea914c0484ebe99e5bf0e4e6f05 in term 0.
I20260810 07:29:18.483744 12175 raft_consensus.cc:2454] T 10310b7d38534687b37ead375be7b64c P b43516e2cada4e9da741b39ff9232154 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7fdcea914c0484ebe99e5bf0e4e6f05 in term 0.
I20260810 07:29:18.484334 11844 leader_election.cc:302] T 10310b7d38534687b37ead375be7b64c P f7fdcea914c0484ebe99e5bf0e4e6f05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b43516e2cada4e9da741b39ff9232154, f7fdcea914c0484ebe99e5bf0e4e6f05; no voters:
I20260810 07:29:18.484679 12283 raft_consensus.cc:2781] T 10310b7d38534687b37ead375be7b64c P f7fdcea914c0484ebe99e5bf0e4e6f05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:29:18.484776 12283 raft_consensus.cc:482] T 10310b7d38534687b37ead375be7b64c P f7fdcea914c0484ebe99e5bf0e4e6f05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:29:18.484807 12283 raft_consensus.cc:3037] T 10310b7d38534687b37ead375be7b64c P f7fdcea914c0484ebe99e5bf0e4e6f05 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:18.485986 12283 raft_consensus.cc:504] T 10310b7d38534687b37ead375be7b64c P f7fdcea914c0484ebe99e5bf0e4e6f05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b43516e2cada4e9da741b39ff9232154" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34563 } } peers { permanent_uuid: "05fc57b1c97349e9916c660e5760b5dc" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 38889 } } peers { permanent_uuid: "f7fdcea914c0484ebe99e5bf0e4e6f05" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 45889 } }
I20260810 07:29:18.486404 12283 leader_election.cc:288] T 10310b7d38534687b37ead375be7b64c P f7fdcea914c0484ebe99e5bf0e4e6f05 [CANDIDATE]: Term 1 election: Requested vote from peers b43516e2cada4e9da741b39ff9232154 (127.12.157.195:34563), 05fc57b1c97349e9916c660e5760b5dc (127.12.157.193:38889)
I20260810 07:29:18.486797 11734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10310b7d38534687b37ead375be7b64c" candidate_uuid: "f7fdcea914c0484ebe99e5bf0e4e6f05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05fc57b1c97349e9916c660e5760b5dc"
I20260810 07:29:18.486897 12175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10310b7d38534687b37ead375be7b64c" candidate_uuid: "f7fdcea914c0484ebe99e5bf0e4e6f05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b43516e2cada4e9da741b39ff9232154"
I20260810 07:29:18.486953 11734 raft_consensus.cc:3037] T 10310b7d38534687b37ead375be7b64c P 05fc57b1c97349e9916c660e5760b5dc [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:18.487088 12175 raft_consensus.cc:3037] T 10310b7d38534687b37ead375be7b64c P b43516e2cada4e9da741b39ff9232154 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:18.488090 11734 raft_consensus.cc:2454] T 10310b7d38534687b37ead375be7b64c P 05fc57b1c97349e9916c660e5760b5dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7fdcea914c0484ebe99e5bf0e4e6f05 in term 1.
I20260810 07:29:18.488247 12175 raft_consensus.cc:2454] T 10310b7d38534687b37ead375be7b64c P b43516e2cada4e9da741b39ff9232154 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7fdcea914c0484ebe99e5bf0e4e6f05 in term 1.
I20260810 07:29:18.488446 11847 leader_election.cc:302] T 10310b7d38534687b37ead375be7b64c P f7fdcea914c0484ebe99e5bf0e4e6f05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05fc57b1c97349e9916c660e5760b5dc, f7fdcea914c0484ebe99e5bf0e4e6f05; no voters:
I20260810 07:29:18.488636 12283 raft_consensus.cc:2781] T 10310b7d38534687b37ead375be7b64c P f7fdcea914c0484ebe99e5bf0e4e6f05 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:18.488898 12283 raft_consensus.cc:686] T 10310b7d38534687b37ead375be7b64c P f7fdcea914c0484ebe99e5bf0e4e6f05 [term 1 LEADER]: Becoming Leader. State: Replica: f7fdcea914c0484ebe99e5bf0e4e6f05, State: Running, Role: LEADER
I20260810 07:29:18.489337 12283 consensus_queue.cc:237] T 10310b7d38534687b37ead375be7b64c P f7fdcea914c0484ebe99e5bf0e4e6f05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b43516e2cada4e9da741b39ff9232154" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34563 } } peers { permanent_uuid: "05fc57b1c97349e9916c660e5760b5dc" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 38889 } } peers { permanent_uuid: "f7fdcea914c0484ebe99e5bf0e4e6f05" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 45889 } }
I20260810 07:29:18.492676 11496 catalog_manager.cc:5495] T 10310b7d38534687b37ead375be7b64c P f7fdcea914c0484ebe99e5bf0e4e6f05 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7fdcea914c0484ebe99e5bf0e4e6f05 (127.12.157.194). New cstate: current_term: 1 leader_uuid: "f7fdcea914c0484ebe99e5bf0e4e6f05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b43516e2cada4e9da741b39ff9232154" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05fc57b1c97349e9916c660e5760b5dc" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 38889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7fdcea914c0484ebe99e5bf0e4e6f05" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 45889 } health_report { overall_health: HEALTHY } } }
I20260810 07:29:18.506102 12265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:18.502414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40066 (local address 127.12.157.194:45889)
0810 07:29:18.502542 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:18.502548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:18.502563 (+ 15us) server_negotiation.cc:407] Connection header received
0810 07:29:18.502681 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:18.502684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:18.502741 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:18.502861 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:29:18.503147 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.503656 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.504547 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.504767 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.504881 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:18.505242 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:29:18.505693 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:29:18.505832 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:18.505893 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 07:29:18.505953 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 07:29:18.513028 12175 raft_consensus.cc:1261] T 10310b7d38534687b37ead375be7b64c P b43516e2cada4e9da741b39ff9232154 [term 1 FOLLOWER]: Refusing update from remote peer f7fdcea914c0484ebe99e5bf0e4e6f05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:29:18.513028 11734 raft_consensus.cc:1261] T 10310b7d38534687b37ead375be7b64c P 05fc57b1c97349e9916c660e5760b5dc [term 1 FOLLOWER]: Refusing update from remote peer f7fdcea914c0484ebe99e5bf0e4e6f05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:29:18.514129 12283 consensus_queue.cc:1035] T 10310b7d38534687b37ead375be7b64c P f7fdcea914c0484ebe99e5bf0e4e6f05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05fc57b1c97349e9916c660e5760b5dc" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 38889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:29:18.514381 12288 consensus_queue.cc:1035] T 10310b7d38534687b37ead375be7b64c P f7fdcea914c0484ebe99e5bf0e4e6f05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b43516e2cada4e9da741b39ff9232154" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:29:18.532994 11505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:29:18.533208 11505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:29:18.535918 11505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 894592c183754455bd16d9ce898b00ab: Sending DeleteTablet for 3 replicas of tablet 10310b7d38534687b37ead375be7b64c
I20260810 07:29:18.536785 11708 tablet_service.cc:1514] Processing DeleteTablet for tablet 10310b7d38534687b37ead375be7b64c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:29:18 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:51936
I20260810 07:29:18.536981 12142 tablet_service.cc:1514] Processing DeleteTablet for tablet 10310b7d38534687b37ead375be7b64c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:29:18 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:59676
I20260810 07:29:18.537120 11925 tablet_service.cc:1514] Processing DeleteTablet for tablet 10310b7d38534687b37ead375be7b64c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:29:18 UTC) from {username='kudu', principal='kudu/127.12.157.254@KRBTEST.COM'} at 127.0.0.1:40058
I20260810 07:29:18.537659 12305 tablet_replica.cc:330] T 10310b7d38534687b37ead375be7b64c P b43516e2cada4e9da741b39ff9232154: stopping tablet replica
I20260810 07:29:18.537784 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 11609
I20260810 07:29:18.538084 12305 raft_consensus.cc:2229] T 10310b7d38534687b37ead375be7b64c P b43516e2cada4e9da741b39ff9232154 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:29:18.538672 12305 raft_consensus.cc:2258] T 10310b7d38534687b37ead375be7b64c P b43516e2cada4e9da741b39ff9232154 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:29:18.541950 12305 ts_tablet_manager.cc:1905] T 10310b7d38534687b37ead375be7b64c P b43516e2cada4e9da741b39ff9232154: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:29:18.545050 12305 ts_tablet_manager.cc:1918] T 10310b7d38534687b37ead375be7b64c P b43516e2cada4e9da741b39ff9232154: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:29:18.545238 12305 log.cc:1196] T 10310b7d38534687b37ead375be7b64c P b43516e2cada4e9da741b39ff9232154: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786346694975122-12919-0/minicluster-data/ts-2/wal/wals/10310b7d38534687b37ead375be7b64c
I20260810 07:29:18.545691 12305 ts_tablet_manager.cc:1939] T 10310b7d38534687b37ead375be7b64c P b43516e2cada4e9da741b39ff9232154: Deleting consensus metadata
I20260810 07:29:18.547268 12306 tablet_replica.cc:330] T 10310b7d38534687b37ead375be7b64c P f7fdcea914c0484ebe99e5bf0e4e6f05: stopping tablet replica
I20260810 07:29:18.547644 12306 raft_consensus.cc:2229] T 10310b7d38534687b37ead375be7b64c P f7fdcea914c0484ebe99e5bf0e4e6f05 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:29:18.548143 12306 raft_consensus.cc:2258] T 10310b7d38534687b37ead375be7b64c P f7fdcea914c0484ebe99e5bf0e4e6f05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:29:18.548319 11477 catalog_manager.cc:4841] TS b43516e2cada4e9da741b39ff9232154 (127.12.157.195:34563): tablet 10310b7d38534687b37ead375be7b64c (table test-table [id=c2fa34efd3104878bf8ab1ca038fea4b]) successfully deleted
W20260810 07:29:18.553951 11478 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.157.193:38889 (error 108)
I20260810 07:29:18.554080 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 11818
I20260810 07:29:18.561510 12261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:18.561193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:38889 (local address 127.0.0.1:51944)
0810 07:29:18.561340 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:18.561422 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.193:38889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 07:29:18.561754 11478 catalog_manager.cc:4584] TS 05fc57b1c97349e9916c660e5760b5dc (127.12.157.193:38889): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 10310b7d38534687b37ead375be7b64c: Network error: Client connection negotiation failed: client connection to 127.12.157.193:38889: connect: Connection refused (error 111)
I20260810 07:29:18.567922 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 12036
I20260810 07:29:18.579530 12261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:18.568204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:45889 (local address 127.0.0.1:40082)
0810 07:29:18.579284 (+ 11080us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:18.579391 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.194:45889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10925}
I20260810 07:29:18.585945 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 11446
2026-08-10T07:29:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2702 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T07:29:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:18Z Disabled control of system clock
I20260810 07:29:18.627748 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:44029
--webserver_interface=127.12.157.254
--webserver_port=0
--builtin_ntp_servers=127.12.157.212:41491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:44029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 07:29:18.741369 12315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:18.741617 12315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:18.741653 12315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:18.745751 12315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:18.745818 12315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:18.745836 12315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:18.745851 12315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:18.745867 12315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:18.750572 12315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:41491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:44029
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:44029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:18.751600 12315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:18.753029 12315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:18.761508 12324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:18.761511 12323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:18.761513 12328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:18.761580 12315 server_base.cc:1034] running on GCE node
I20260810 07:29:18.762331 12315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:18.763407 12315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:18.764618 12315 hybrid_clock.cc:648] HybridClock initialized: now 1786346958764593 us; error 39 us; skew 500 ppm
I20260810 07:29:18.769464 12315 webserver.cc:460] Webserver started at http://127.12.157.254:39307/ using document root <none> and password file <none>
I20260810 07:29:18.770064 12315 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:18.770128 12315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:18.770370 12315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:18.772445 12315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "51cde13322fe4bdea9f23b9d665ef718"
format_stamp: "Formatted at 2026-08-10 07:29:18 on dist-test-slave-ffrd"
server_key: "2897ffeb476ee6f244f0c99e160543b0"
server_key_iv: "faa75f6c79fc4f65351cc1042cd63ee2"
server_key_version: "encryptionkey@0"
I20260810 07:29:18.773103 12315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "51cde13322fe4bdea9f23b9d665ef718"
format_stamp: "Formatted at 2026-08-10 07:29:18 on dist-test-slave-ffrd"
server_key: "2897ffeb476ee6f244f0c99e160543b0"
server_key_iv: "faa75f6c79fc4f65351cc1042cd63ee2"
server_key_version: "encryptionkey@0"
I20260810 07:29:18.776921 12315 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 07:29:18.779800 12338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:18.781494 12315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 07:29:18.781641 12315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "51cde13322fe4bdea9f23b9d665ef718"
format_stamp: "Formatted at 2026-08-10 07:29:18 on dist-test-slave-ffrd"
server_key: "2897ffeb476ee6f244f0c99e160543b0"
server_key_iv: "faa75f6c79fc4f65351cc1042cd63ee2"
server_key_version: "encryptionkey@0"
I20260810 07:29:18.781805 12315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:18.798585 12315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:18.799305 12315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:18.799513 12315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:18.807938 12315 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:44029
I20260810 07:29:18.807960 12428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:44029 every 8 connection(s)
I20260810 07:29:18.809027 12315 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:18.812314 12429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:18.814220 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 12315
I20260810 07:29:18.814378 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:29:18.814690 12919 external_mini_cluster.cc:1428] Setting key 02bdd5c16d44ccd86edae3b43c2f699a
I20260810 07:29:18.819454 12429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51cde13322fe4bdea9f23b9d665ef718: Bootstrap starting.
I20260810 07:29:18.822419 12429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51cde13322fe4bdea9f23b9d665ef718: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:18.823295 12429 log.cc:824] T 00000000000000000000000000000000 P 51cde13322fe4bdea9f23b9d665ef718: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:18.824569 12437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:18.816197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55364 (local address 127.12.157.254:44029)
0810 07:29:18.816843 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:18.816860 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:18.816922 (+ 62us) server_negotiation.cc:407] Connection header received
0810 07:29:18.817643 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:18.817660 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:18.818045 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:18.818326 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:18.818675 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.819700 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.821000 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:18.821493 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:18.821617 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:18.822573 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:18.822586 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:18.822599 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:18.823030 (+ 431us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:18.823087 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:18.823108 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:18.823185 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:18.823356 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 07:29:18.823635 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":153,"threads_started":1}
I20260810 07:29:18.825737 12429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51cde13322fe4bdea9f23b9d665ef718: No bootstrap required, opened a new log
I20260810 07:29:18.829715 12429 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51cde13322fe4bdea9f23b9d665ef718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cde13322fe4bdea9f23b9d665ef718" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44029 } }
I20260810 07:29:18.829929 12429 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51cde13322fe4bdea9f23b9d665ef718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:18.829964 12429 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51cde13322fe4bdea9f23b9d665ef718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51cde13322fe4bdea9f23b9d665ef718, State: Initialized, Role: FOLLOWER
I20260810 07:29:18.830380 12429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51cde13322fe4bdea9f23b9d665ef718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cde13322fe4bdea9f23b9d665ef718" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44029 } }
I20260810 07:29:18.830547 12429 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51cde13322fe4bdea9f23b9d665ef718 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:29:18.830636 12429 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51cde13322fe4bdea9f23b9d665ef718 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:29:18.830742 12429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51cde13322fe4bdea9f23b9d665ef718 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:18.831723 12429 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51cde13322fe4bdea9f23b9d665ef718 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cde13322fe4bdea9f23b9d665ef718" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44029 } }
I20260810 07:29:18.832069 12429 leader_election.cc:302] T 00000000000000000000000000000000 P 51cde13322fe4bdea9f23b9d665ef718 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51cde13322fe4bdea9f23b9d665ef718; no voters:
I20260810 07:29:18.832427 12429 leader_election.cc:288] T 00000000000000000000000000000000 P 51cde13322fe4bdea9f23b9d665ef718 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:29:18.832568 12442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51cde13322fe4bdea9f23b9d665ef718 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:18.832861 12442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51cde13322fe4bdea9f23b9d665ef718 [term 1 LEADER]: Becoming Leader. State: Replica: 51cde13322fe4bdea9f23b9d665ef718, State: Running, Role: LEADER
I20260810 07:29:18.833222 12442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51cde13322fe4bdea9f23b9d665ef718 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cde13322fe4bdea9f23b9d665ef718" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44029 } }
I20260810 07:29:18.833739 12429 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51cde13322fe4bdea9f23b9d665ef718 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:18.834941 12445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51cde13322fe4bdea9f23b9d665ef718 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51cde13322fe4bdea9f23b9d665ef718. Latest consensus state: current_term: 1 leader_uuid: "51cde13322fe4bdea9f23b9d665ef718" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cde13322fe4bdea9f23b9d665ef718" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44029 } } }
I20260810 07:29:18.835075 12445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51cde13322fe4bdea9f23b9d665ef718 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:18.835381 12444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51cde13322fe4bdea9f23b9d665ef718 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51cde13322fe4bdea9f23b9d665ef718" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cde13322fe4bdea9f23b9d665ef718" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 44029 } } }
I20260810 07:29:18.835566 12444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51cde13322fe4bdea9f23b9d665ef718 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:18.835475 12456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:18.839246 12456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:18.844589 12456 catalog_manager.cc:1347] Generated new cluster ID: 471867c866c84283a0d8ad17c0e79028
I20260810 07:29:18.844717 12456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:18.856875 12456 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:18.858152 12456 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:18.865490 12456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51cde13322fe4bdea9f23b9d665ef718: Generated new TSK 0
I20260810 07:29:18.866217 12456 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:29:18.870399 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.193:0
--local_ip_for_outbound_sockets=127.12.157.193
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:44029
--builtin_ntp_servers=127.12.157.212:41491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 07:29:18.993728 12472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:18.994032 12472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:18.994103 12472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:18.998185 12472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:18.998257 12472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:18.998346 12472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.193
I20260810 07:29:19.003046 12472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:41491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.157.193
--webserver_port=0
--tserver_master_addrs=127.12.157.254:44029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.193
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:19.004169 12472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:19.005609 12472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:19.015681 12481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:19.015705 12484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:19.015808 12472 server_base.cc:1034] running on GCE node
W20260810 07:29:19.015681 12487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:19.016402 12472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:19.017073 12472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:19.018253 12472 hybrid_clock.cc:648] HybridClock initialized: now 1786346959018217 us; error 30 us; skew 500 ppm
I20260810 07:29:19.020134 12472 webserver.cc:460] Webserver started at http://127.12.157.193:46125/ using document root <none> and password file <none>
I20260810 07:29:19.020673 12472 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:19.020730 12472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:19.020905 12472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:19.022980 12472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-0/data/instance:
uuid: "86cfc168c7df4b6ba49c45f12e9b9baa"
format_stamp: "Formatted at 2026-08-10 07:29:19 on dist-test-slave-ffrd"
server_key: "06d686b253669f4c5af38be3dc8002da"
server_key_iv: "b2853127812a5527b85cdc790a2a8b5a"
server_key_version: "encryptionkey@0"
I20260810 07:29:19.023489 12472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance:
uuid: "86cfc168c7df4b6ba49c45f12e9b9baa"
format_stamp: "Formatted at 2026-08-10 07:29:19 on dist-test-slave-ffrd"
server_key: "06d686b253669f4c5af38be3dc8002da"
server_key_iv: "b2853127812a5527b85cdc790a2a8b5a"
server_key_version: "encryptionkey@0"
I20260810 07:29:19.027269 12472 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 07:29:19.029980 12494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:19.031307 12472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 07:29:19.031414 12472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-0/wal
uuid: "86cfc168c7df4b6ba49c45f12e9b9baa"
format_stamp: "Formatted at 2026-08-10 07:29:19 on dist-test-slave-ffrd"
server_key: "06d686b253669f4c5af38be3dc8002da"
server_key_iv: "b2853127812a5527b85cdc790a2a8b5a"
server_key_version: "encryptionkey@0"
I20260810 07:29:19.031507 12472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:19.052824 12472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:19.053638 12472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:19.053807 12472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:19.054392 12472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:29:19.055367 12472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:29:19.055416 12472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:19.055460 12472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:29:19.055475 12472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:19.065405 12472 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.193:43859
I20260810 07:29:19.065439 12683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.193:43859 every 8 connection(s)
I20260810 07:29:19.066355 12472 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-0/data/info.pb
I20260810 07:29:19.068761 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 12472
I20260810 07:29:19.068924 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-0/wal/instance
I20260810 07:29:19.069308 12919 external_mini_cluster.cc:1428] Setting key 2cfcac98794cb56670d9a1c9f6aa28f0
I20260810 07:29:19.071821 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.194:0
--local_ip_for_outbound_sockets=127.12.157.194
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:44029
--builtin_ntp_servers=127.12.157.212:41491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 07:29:19.077529 12437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:19.068276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.193:58277 (local address 127.12.157.254:44029)
0810 07:29:19.068445 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:19.068450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:19.069919 (+ 1469us) server_negotiation.cc:407] Connection header received
0810 07:29:19.071691 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:19.071695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:19.071747 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:19.071834 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:19.073146 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.073706 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.075196 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.075444 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.075562 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:19.076989 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:19.076992 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:19.076994 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:19.077008 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:19.077028 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:19.077036 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:19.077119 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:19.077295 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 07:29:19.077379 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 07:29:19.078445 12690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:19.068714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:44029 (local address 127.12.157.193:58277)
0810 07:29:19.069718 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:19.069779 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:19.071410 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:19.072035 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:19.072046 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:19.072162 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:19.072971 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:19.072997 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.073853 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.073857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:19.075049 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:19.075060 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.075608 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.075612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:19.075790 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:19.076587 (+ 797us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:29:19.076588 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:19.076632 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:29:19.076699 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:19.077135 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:19.077146 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:19.077154 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 07:29:19.077292 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 07:29:19.077607 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":194,"threads_started":1}
I20260810 07:29:19.079815 12685 heartbeater.cc:347] Connected to a master server at 127.12.157.254:44029
I20260810 07:29:19.080078 12685 heartbeater.cc:464] Registering TS with master...
I20260810 07:29:19.080626 12685 heartbeater.cc:511] Master 127.12.157.254:44029 requested a full tablet report, sending...
I20260810 07:29:19.082516 12367 ts_manager.cc:194] Registered new tserver with Master: 86cfc168c7df4b6ba49c45f12e9b9baa (127.12.157.193:43859)
I20260810 07:29:19.084327 12367 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.157.193:58277
W20260810 07:29:19.191828 12691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:19.192096 12691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:19.192135 12691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:19.196132 12691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:19.196193 12691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:19.196249 12691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.194
I20260810 07:29:19.200860 12691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:41491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.157.194
--webserver_port=0
--tserver_master_addrs=127.12.157.254:44029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.194
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:19.201975 12691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:19.203312 12691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:19.213125 12713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:19.213135 12700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:19.213248 12691 server_base.cc:1034] running on GCE node
W20260810 07:29:19.213135 12698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:19.213881 12691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:19.214556 12691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:19.215790 12691 hybrid_clock.cc:648] HybridClock initialized: now 1786346959215763 us; error 47 us; skew 500 ppm
I20260810 07:29:19.217864 12691 webserver.cc:460] Webserver started at http://127.12.157.194:43745/ using document root <none> and password file <none>
I20260810 07:29:19.218487 12691 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:19.218590 12691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:19.218847 12691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:19.221125 12691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-1/data/instance:
uuid: "1463aabdbc9c46abbcff8fe150be4a5c"
format_stamp: "Formatted at 2026-08-10 07:29:19 on dist-test-slave-ffrd"
server_key: "d201bb6e9d296135fcf80ef36980fb7f"
server_key_iv: "7bf27d04244a07c062ae1d7b688b2705"
server_key_version: "encryptionkey@0"
I20260810 07:29:19.221756 12691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance:
uuid: "1463aabdbc9c46abbcff8fe150be4a5c"
format_stamp: "Formatted at 2026-08-10 07:29:19 on dist-test-slave-ffrd"
server_key: "d201bb6e9d296135fcf80ef36980fb7f"
server_key_iv: "7bf27d04244a07c062ae1d7b688b2705"
server_key_version: "encryptionkey@0"
I20260810 07:29:19.225680 12691 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 07:29:19.228327 12721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:19.229760 12691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:29:19.229915 12691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-1/wal
uuid: "1463aabdbc9c46abbcff8fe150be4a5c"
format_stamp: "Formatted at 2026-08-10 07:29:19 on dist-test-slave-ffrd"
server_key: "d201bb6e9d296135fcf80ef36980fb7f"
server_key_iv: "7bf27d04244a07c062ae1d7b688b2705"
server_key_version: "encryptionkey@0"
I20260810 07:29:19.230038 12691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:19.252784 12691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:19.253597 12691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:19.253810 12691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:19.254446 12691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:29:19.255434 12691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:29:19.255505 12691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:19.255578 12691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:29:19.255630 12691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:19.265486 12915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.194:41753 every 8 connection(s)
I20260810 07:29:19.265527 12691 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.194:41753
I20260810 07:29:19.266520 12691 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-1/data/info.pb
I20260810 07:29:19.270529 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 12691
I20260810 07:29:19.270650 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-1/wal/instance
I20260810 07:29:19.270994 12919 external_mini_cluster.cc:1428] Setting key f82b9144b7034b1fd6d224d943aad155
I20260810 07:29:19.273554 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.157.195:0
--local_ip_for_outbound_sockets=127.12.157.195
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:44029
--builtin_ntp_servers=127.12.157.212:41491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 07:29:19.277622 12437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:19.268695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.194:45441 (local address 127.12.157.254:44029)
0810 07:29:19.268867 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:19.268872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:19.270073 (+ 1201us) server_negotiation.cc:407] Connection header received
0810 07:29:19.271870 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:19.271874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:19.271929 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:19.272034 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:19.273253 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.274097 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.275480 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.275711 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.275823 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:19.277078 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:19.277082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:19.277085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:19.277107 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:19.277128 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:19.277136 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:19.277219 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:19.277399 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 07:29:19.277473 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 07:29:19.278538 12927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:19.269170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:44029 (local address 127.12.157.194:45441)
0810 07:29:19.269907 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:19.269955 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:19.271606 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:19.272188 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:19.272194 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:19.272282 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:19.273091 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:19.273110 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.274244 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.274247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:19.275299 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:19.275310 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.275881 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.275886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:19.276056 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:19.276740 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:29:19.276742 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:19.276784 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:29:19.276857 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:19.277223 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:19.277236 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:19.277243 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 07:29:19.277387 (+ 144us) client_negotiation.cc:240] Negotiation successful
0810 07:29:19.277696 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":138,"threads_started":1}
I20260810 07:29:19.279868 12916 heartbeater.cc:347] Connected to a master server at 127.12.157.254:44029
I20260810 07:29:19.280138 12916 heartbeater.cc:464] Registering TS with master...
I20260810 07:29:19.280706 12916 heartbeater.cc:511] Master 127.12.157.254:44029 requested a full tablet report, sending...
I20260810 07:29:19.282035 12367 ts_manager.cc:194] Registered new tserver with Master: 1463aabdbc9c46abbcff8fe150be4a5c (127.12.157.194:41753)
I20260810 07:29:19.282950 12367 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.157.194:45441
W20260810 07:29:19.393347 12931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:19.393592 12931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:19.393630 12931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:19.397603 12931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:19.397657 12931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:19.397711 12931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.157.195
I20260810 07:29:19.402386 12931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:41491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.157.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.157.195
--webserver_port=0
--tserver_master_addrs=127.12.157.254:44029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.157.195
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:19.403375 12931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:19.404667 12931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 07:29:19.414438 12931 server_base.cc:1034] running on GCE node
W20260810 07:29:19.414376 12939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:19.414386 12938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:19.414362 12941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:19.414954 12931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:19.415568 12931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:19.416764 12931 hybrid_clock.cc:648] HybridClock initialized: now 1786346959416749 us; error 27 us; skew 500 ppm
I20260810 07:29:19.418685 12931 webserver.cc:460] Webserver started at http://127.12.157.195:39551/ using document root <none> and password file <none>
I20260810 07:29:19.419245 12931 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:19.419339 12931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:19.419577 12931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:19.421653 12931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-2/data/instance:
uuid: "77238a9fb1fd4965ac124fd5cdce98c9"
format_stamp: "Formatted at 2026-08-10 07:29:19 on dist-test-slave-ffrd"
server_key: "65cf2ea1b011286b7690177fbc70d070"
server_key_iv: "7f1072572936402aee0552c62e3fe0a5"
server_key_version: "encryptionkey@0"
I20260810 07:29:19.422219 12931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance:
uuid: "77238a9fb1fd4965ac124fd5cdce98c9"
format_stamp: "Formatted at 2026-08-10 07:29:19 on dist-test-slave-ffrd"
server_key: "65cf2ea1b011286b7690177fbc70d070"
server_key_iv: "7f1072572936402aee0552c62e3fe0a5"
server_key_version: "encryptionkey@0"
I20260810 07:29:19.425851 12931 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 07:29:19.428237 12956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:19.429633 12931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:29:19.429764 12931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-2/wal
uuid: "77238a9fb1fd4965ac124fd5cdce98c9"
format_stamp: "Formatted at 2026-08-10 07:29:19 on dist-test-slave-ffrd"
server_key: "65cf2ea1b011286b7690177fbc70d070"
server_key_iv: "7f1072572936402aee0552c62e3fe0a5"
server_key_version: "encryptionkey@0"
I20260810 07:29:19.429874 12931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:19.448128 12931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:19.448815 12931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:19.449075 12931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:19.449682 12931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 07:29:19.450649 12931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 07:29:19.450722 12931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:19.450791 12931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 07:29:19.450838 12931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:19.460145 12931 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.195:41275
I20260810 07:29:19.460184 13143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.195:41275 every 8 connection(s)
I20260810 07:29:19.461061 12931 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-2/data/info.pb
I20260810 07:29:19.461545 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 12931
I20260810 07:29:19.461668 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-2/wal/instance
I20260810 07:29:19.461988 12919 external_mini_cluster.cc:1428] Setting key 4fe5048b9a3b02415cba3d55965afa5a
I20260810 07:29:19.470302 12437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:19.463065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:43771 (local address 127.12.157.254:44029)
0810 07:29:19.463200 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:19.463207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:19.464442 (+ 1235us) server_negotiation.cc:407] Connection header received
0810 07:29:19.465884 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:19.465887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:19.465933 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:19.466002 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:19.466831 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.467345 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.468526 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.468728 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.468802 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:19.469900 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:19.469903 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:19.469905 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:19.469917 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:19.469934 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:19.469942 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:19.470000 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:19.470149 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 07:29:19.470196 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 07:29:19.471123 13149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:19.463499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:44029 (local address 127.12.157.195:43771)
0810 07:29:19.464267 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:19.464307 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:19.465702 (+ 1395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:19.466158 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:19.466165 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:19.466242 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:19.466712 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:19.466724 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.467494 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.467497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:19.468412 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:19.468423 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.468856 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.468859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:19.469072 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:19.469654 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 07:29:19.469656 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:19.469693 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 07:29:19.469757 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:19.470016 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:19.470023 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:19.470027 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 07:29:19.470129 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 07:29:19.470378 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":147,"threads_started":1}
I20260810 07:29:19.472301 13145 heartbeater.cc:347] Connected to a master server at 127.12.157.254:44029
I20260810 07:29:19.472561 13145 heartbeater.cc:464] Registering TS with master...
I20260810 07:29:19.473140 13145 heartbeater.cc:511] Master 127.12.157.254:44029 requested a full tablet report, sending...
I20260810 07:29:19.474172 12367 ts_manager.cc:194] Registered new tserver with Master: 77238a9fb1fd4965ac124fd5cdce98c9 (127.12.157.195:41275)
I20260810 07:29:19.474735 12367 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.157.195:43771
I20260810 07:29:19.476194 12919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 07:29:19.482175 12437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:19.478203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55374 (local address 127.12.157.254:44029)
0810 07:29:19.478369 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:19.478374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:19.478525 (+ 151us) server_negotiation.cc:407] Connection header received
0810 07:29:19.478676 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:19.478681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:19.478730 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:19.478814 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:19.479129 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.479632 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.480502 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.480716 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.480826 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:19.481695 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:19.481699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:19.481700 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:19.481712 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:19.481730 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:19.481738 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:19.481818 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:19.481963 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 07:29:19.482020 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 07:29:19.486101 12361 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 07:29:19.490589 12361 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 07:29:19.508975 13162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:19.502671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.195:41275 (local address 127.0.0.1:37592)
0810 07:29:19.503890 (+ 1219us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:19.503900 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:19.503961 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:19.504890 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:19.504896 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:19.504914 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:19.505091 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:19.505099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.506552 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.506556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:19.507486 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:19.507496 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.508357 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.508360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:19.508469 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:19.508507 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 07:29:19.508632 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 07:29:19.508772 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1152,"mutex_wait_us":59,"thread_start_us":121,"threads_started":1}
I20260810 07:29:19.508975 13170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:19.503545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37592 (local address 127.12.157.195:41275)
0810 07:29:19.504459 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:19.504470 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:19.504492 (+ 22us) server_negotiation.cc:407] Connection header received
0810 07:29:19.504564 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:19.504568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:19.504731 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:19.505016 (+ 285us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:19.505202 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.506434 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.507626 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.508214 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.508369 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:19.508571 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:19.508663 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 07:29:19.508765 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":812,"thread_start_us":181,"threads_started":1}
I20260810 07:29:19.510878 13163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:19.502590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:43859 (local address 127.0.0.1:51784)
0810 07:29:19.503474 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:19.503530 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:19.503660 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:19.504722 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:19.504728 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:19.504761 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:19.504984 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:19.504995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.506320 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.506324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:19.508845 (+ 2521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:19.508855 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.510551 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.510555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:19.510636 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:19.510656 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 07:29:19.510717 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 07:29:19.510770 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":981,"thread_start_us":100,"threads_started":1}
I20260810 07:29:19.511616 13168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:19.503633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:41753 (local address 127.0.0.1:50354)
0810 07:29:19.504076 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:19.504090 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:19.504185 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:19.505064 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:19.505069 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:19.505090 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:19.505198 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:19.505205 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.506612 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.506615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:19.507486 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:19.507496 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.511186 (+ 3690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.511190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:19.511298 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:19.511322 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 07:29:19.511413 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 07:29:19.511492 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":28,"thread_start_us":118,"threads_started":1}
I20260810 07:29:19.512120 13172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:19.504105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50354 (local address 127.12.157.194:41753)
0810 07:29:19.504439 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:19.504466 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:19.504492 (+ 26us) server_negotiation.cc:407] Connection header received
0810 07:29:19.504557 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:19.504560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:19.504738 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:19.504889 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:19.505340 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.506476 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.507627 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.508736 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.508864 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:19.511810 (+ 2946us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:19.511863 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 07:29:19.511959 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":135,"threads_started":1}
I20260810 07:29:19.512405 13164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:19.502709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51784 (local address 127.12.157.193:43859)
0810 07:29:19.504190 (+ 1481us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:19.504202 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:19.504229 (+ 27us) server_negotiation.cc:407] Connection header received
0810 07:29:19.504315 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:19.504320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:19.504541 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:19.504707 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:19.505132 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.506188 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.509414 (+ 3226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.510292 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.510576 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:19.510739 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:19.510791 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 07:29:19.510882 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1365,"thread_start_us":166,"threads_started":1}
I20260810 07:29:19.512816 13039 tablet_service.cc:1467] Processing CreateTablet for tablet 3c978e0e91a348ecb03253b417e070a2 (DEFAULT_TABLE table=test-table [id=e65bc4bd06c94f1abcc4adaff76e4f9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:29:19.513993 13039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c978e0e91a348ecb03253b417e070a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:19.515265 12566 tablet_service.cc:1467] Processing CreateTablet for tablet 3c978e0e91a348ecb03253b417e070a2 (DEFAULT_TABLE table=test-table [id=e65bc4bd06c94f1abcc4adaff76e4f9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:29:19.515265 12804 tablet_service.cc:1467] Processing CreateTablet for tablet 3c978e0e91a348ecb03253b417e070a2 (DEFAULT_TABLE table=test-table [id=e65bc4bd06c94f1abcc4adaff76e4f9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 07:29:19.516469 12566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c978e0e91a348ecb03253b417e070a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:19.516469 12804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c978e0e91a348ecb03253b417e070a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:19.520721 13176 tablet_bootstrap.cc:492] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9: Bootstrap starting.
I20260810 07:29:19.523423 13176 tablet_bootstrap.cc:654] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:19.524109 13183 tablet_bootstrap.cc:492] T 3c978e0e91a348ecb03253b417e070a2 P 1463aabdbc9c46abbcff8fe150be4a5c: Bootstrap starting.
I20260810 07:29:19.524641 13176 log.cc:824] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:19.527038 13176 tablet_bootstrap.cc:492] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9: No bootstrap required, opened a new log
I20260810 07:29:19.527259 13183 tablet_bootstrap.cc:654] T 3c978e0e91a348ecb03253b417e070a2 P 1463aabdbc9c46abbcff8fe150be4a5c: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:19.527262 13176 ts_tablet_manager.cc:1397] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 07:29:19.528206 13183 log.cc:824] T 3c978e0e91a348ecb03253b417e070a2 P 1463aabdbc9c46abbcff8fe150be4a5c: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:19.529139 13182 tablet_bootstrap.cc:492] T 3c978e0e91a348ecb03253b417e070a2 P 86cfc168c7df4b6ba49c45f12e9b9baa: Bootstrap starting.
I20260810 07:29:19.531116 13182 tablet_bootstrap.cc:654] T 3c978e0e91a348ecb03253b417e070a2 P 86cfc168c7df4b6ba49c45f12e9b9baa: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:19.531177 13183 tablet_bootstrap.cc:492] T 3c978e0e91a348ecb03253b417e070a2 P 1463aabdbc9c46abbcff8fe150be4a5c: No bootstrap required, opened a new log
I20260810 07:29:19.531373 13183 ts_tablet_manager.cc:1397] T 3c978e0e91a348ecb03253b417e070a2 P 1463aabdbc9c46abbcff8fe150be4a5c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 07:29:19.531911 13182 log.cc:824] T 3c978e0e91a348ecb03253b417e070a2 P 86cfc168c7df4b6ba49c45f12e9b9baa: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:19.533147 13176 raft_consensus.cc:348] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cfc168c7df4b6ba49c45f12e9b9baa" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 43859 } } peers { permanent_uuid: "77238a9fb1fd4965ac124fd5cdce98c9" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41275 } } peers { permanent_uuid: "1463aabdbc9c46abbcff8fe150be4a5c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 41753 } }
I20260810 07:29:19.533535 13176 raft_consensus.cc:374] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:19.533680 13176 raft_consensus.cc:729] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77238a9fb1fd4965ac124fd5cdce98c9, State: Initialized, Role: FOLLOWER
I20260810 07:29:19.534210 13176 consensus_queue.cc:260] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cfc168c7df4b6ba49c45f12e9b9baa" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 43859 } } peers { permanent_uuid: "77238a9fb1fd4965ac124fd5cdce98c9" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41275 } } peers { permanent_uuid: "1463aabdbc9c46abbcff8fe150be4a5c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 41753 } }
I20260810 07:29:19.534704 13182 tablet_bootstrap.cc:492] T 3c978e0e91a348ecb03253b417e070a2 P 86cfc168c7df4b6ba49c45f12e9b9baa: No bootstrap required, opened a new log
I20260810 07:29:19.534943 13182 ts_tablet_manager.cc:1397] T 3c978e0e91a348ecb03253b417e070a2 P 86cfc168c7df4b6ba49c45f12e9b9baa: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260810 07:29:19.535270 13145 heartbeater.cc:503] Master 127.12.157.254:44029 was elected leader, sending a full tablet report...
I20260810 07:29:19.535490 13176 ts_tablet_manager.cc:1428] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260810 07:29:19.537635 13183 raft_consensus.cc:348] T 3c978e0e91a348ecb03253b417e070a2 P 1463aabdbc9c46abbcff8fe150be4a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cfc168c7df4b6ba49c45f12e9b9baa" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 43859 } } peers { permanent_uuid: "77238a9fb1fd4965ac124fd5cdce98c9" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41275 } } peers { permanent_uuid: "1463aabdbc9c46abbcff8fe150be4a5c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 41753 } }
I20260810 07:29:19.537839 13183 raft_consensus.cc:374] T 3c978e0e91a348ecb03253b417e070a2 P 1463aabdbc9c46abbcff8fe150be4a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:19.537873 13183 raft_consensus.cc:729] T 3c978e0e91a348ecb03253b417e070a2 P 1463aabdbc9c46abbcff8fe150be4a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1463aabdbc9c46abbcff8fe150be4a5c, State: Initialized, Role: FOLLOWER
I20260810 07:29:19.538300 13183 consensus_queue.cc:260] T 3c978e0e91a348ecb03253b417e070a2 P 1463aabdbc9c46abbcff8fe150be4a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cfc168c7df4b6ba49c45f12e9b9baa" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 43859 } } peers { permanent_uuid: "77238a9fb1fd4965ac124fd5cdce98c9" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41275 } } peers { permanent_uuid: "1463aabdbc9c46abbcff8fe150be4a5c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 41753 } }
I20260810 07:29:19.539266 12916 heartbeater.cc:503] Master 127.12.157.254:44029 was elected leader, sending a full tablet report...
I20260810 07:29:19.539340 13183 ts_tablet_manager.cc:1428] T 3c978e0e91a348ecb03253b417e070a2 P 1463aabdbc9c46abbcff8fe150be4a5c: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260810 07:29:19.539288 13182 raft_consensus.cc:348] T 3c978e0e91a348ecb03253b417e070a2 P 86cfc168c7df4b6ba49c45f12e9b9baa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cfc168c7df4b6ba49c45f12e9b9baa" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 43859 } } peers { permanent_uuid: "77238a9fb1fd4965ac124fd5cdce98c9" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41275 } } peers { permanent_uuid: "1463aabdbc9c46abbcff8fe150be4a5c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 41753 } }
I20260810 07:29:19.539533 13182 raft_consensus.cc:374] T 3c978e0e91a348ecb03253b417e070a2 P 86cfc168c7df4b6ba49c45f12e9b9baa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:19.539587 13182 raft_consensus.cc:729] T 3c978e0e91a348ecb03253b417e070a2 P 86cfc168c7df4b6ba49c45f12e9b9baa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86cfc168c7df4b6ba49c45f12e9b9baa, State: Initialized, Role: FOLLOWER
I20260810 07:29:19.540035 13182 consensus_queue.cc:260] T 3c978e0e91a348ecb03253b417e070a2 P 86cfc168c7df4b6ba49c45f12e9b9baa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cfc168c7df4b6ba49c45f12e9b9baa" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 43859 } } peers { permanent_uuid: "77238a9fb1fd4965ac124fd5cdce98c9" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41275 } } peers { permanent_uuid: "1463aabdbc9c46abbcff8fe150be4a5c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 41753 } }
I20260810 07:29:19.540911 13182 ts_tablet_manager.cc:1428] T 3c978e0e91a348ecb03253b417e070a2 P 86cfc168c7df4b6ba49c45f12e9b9baa: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 07:29:19.541327 12685 heartbeater.cc:503] Master 127.12.157.254:44029 was elected leader, sending a full tablet report...
W20260810 07:29:19.567627 12689 tablet.cc:2369] T 3c978e0e91a348ecb03253b417e070a2 P 86cfc168c7df4b6ba49c45f12e9b9baa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 07:29:19.712248 13146 tablet.cc:2369] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 07:29:19.767918 12918 tablet.cc:2369] T 3c978e0e91a348ecb03253b417e070a2 P 1463aabdbc9c46abbcff8fe150be4a5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:29:19.811197 13189 raft_consensus.cc:482] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 07:29:19.811434 13189 raft_consensus.cc:504] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cfc168c7df4b6ba49c45f12e9b9baa" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 43859 } } peers { permanent_uuid: "77238a9fb1fd4965ac124fd5cdce98c9" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41275 } } peers { permanent_uuid: "1463aabdbc9c46abbcff8fe150be4a5c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 41753 } }
I20260810 07:29:19.812852 13189 leader_election.cc:288] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86cfc168c7df4b6ba49c45f12e9b9baa (127.12.157.193:43859), 1463aabdbc9c46abbcff8fe150be4a5c (127.12.157.194:41753)
I20260810 07:29:19.817519 13149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:19.813224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.194:41753 (local address 127.12.157.195:53129)
0810 07:29:19.813449 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:19.813471 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:19.813586 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:19.813955 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:19.813959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:19.813982 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:19.814138 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:19.814144 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.815000 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.815006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:19.816137 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:19.816148 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.817059 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.817063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:19.817167 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:19.817207 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 07:29:19.817279 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 07:29:19.817374 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":1}
I20260810 07:29:19.817613 13172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:19.813563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:53129 (local address 127.12.157.194:41753)
0810 07:29:19.813694 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:19.813699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:19.813713 (+ 14us) server_negotiation.cc:407] Connection header received
0810 07:29:19.813767 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:19.813770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:19.813816 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:19.813898 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:19.814276 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.814886 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.816275 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.816910 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.817048 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:19.817099 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:19.817406 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 07:29:19.817484 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 07:29:19.817780 13164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:19.813438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.157.195:59033 (local address 127.12.157.193:43859)
0810 07:29:19.813876 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:19.813884 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:19.813944 (+ 60us) server_negotiation.cc:407] Connection header received
0810 07:29:19.814120 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:19.814123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:19.814172 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:19.814239 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 07:29:19.814540 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.815297 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.816595 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.817153 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.817288 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:19.817333 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:19.817578 (+ 245us) server_negotiation.cc:299] Negotiation successful
0810 07:29:19.817658 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322}
I20260810 07:29:19.818176 13194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:19.813224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:43859 (local address 127.12.157.195:59033)
0810 07:29:19.813845 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:19.813858 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:19.813996 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:19.814294 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:19.814298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:19.814319 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 07:29:19.814432 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:19.814437 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.815433 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.815436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:19.816474 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:19.816482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.817292 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.817296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:19.817423 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:19.817446 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 07:29:19.817957 (+ 511us) client_negotiation.cc:240] Negotiation successful
0810 07:29:19.818038 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":216,"thread_start_us":126,"threads_started":1}
I20260810 07:29:19.818254 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c978e0e91a348ecb03253b417e070a2" candidate_uuid: "77238a9fb1fd4965ac124fd5cdce98c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1463aabdbc9c46abbcff8fe150be4a5c" is_pre_election: true
I20260810 07:29:19.818580 12843 raft_consensus.cc:2454] T 3c978e0e91a348ecb03253b417e070a2 P 1463aabdbc9c46abbcff8fe150be4a5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77238a9fb1fd4965ac124fd5cdce98c9 in term 0.
I20260810 07:29:19.819113 12606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c978e0e91a348ecb03253b417e070a2" candidate_uuid: "77238a9fb1fd4965ac124fd5cdce98c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86cfc168c7df4b6ba49c45f12e9b9baa" is_pre_election: true
I20260810 07:29:19.819156 12960 leader_election.cc:302] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1463aabdbc9c46abbcff8fe150be4a5c, 77238a9fb1fd4965ac124fd5cdce98c9; no voters:
I20260810 07:29:19.819443 12606 raft_consensus.cc:2454] T 3c978e0e91a348ecb03253b417e070a2 P 86cfc168c7df4b6ba49c45f12e9b9baa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77238a9fb1fd4965ac124fd5cdce98c9 in term 0.
I20260810 07:29:19.819496 13189 raft_consensus.cc:2781] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 07:29:19.819582 13189 raft_consensus.cc:482] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 07:29:19.819635 13189 raft_consensus.cc:3037] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:19.820739 13189 raft_consensus.cc:504] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cfc168c7df4b6ba49c45f12e9b9baa" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 43859 } } peers { permanent_uuid: "77238a9fb1fd4965ac124fd5cdce98c9" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41275 } } peers { permanent_uuid: "1463aabdbc9c46abbcff8fe150be4a5c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 41753 } }
I20260810 07:29:19.821175 13189 leader_election.cc:288] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9 [CANDIDATE]: Term 1 election: Requested vote from peers 86cfc168c7df4b6ba49c45f12e9b9baa (127.12.157.193:43859), 1463aabdbc9c46abbcff8fe150be4a5c (127.12.157.194:41753)
I20260810 07:29:19.821527 12606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c978e0e91a348ecb03253b417e070a2" candidate_uuid: "77238a9fb1fd4965ac124fd5cdce98c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86cfc168c7df4b6ba49c45f12e9b9baa"
I20260810 07:29:19.821584 12843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c978e0e91a348ecb03253b417e070a2" candidate_uuid: "77238a9fb1fd4965ac124fd5cdce98c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1463aabdbc9c46abbcff8fe150be4a5c"
I20260810 07:29:19.821678 12606 raft_consensus.cc:3037] T 3c978e0e91a348ecb03253b417e070a2 P 86cfc168c7df4b6ba49c45f12e9b9baa [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:19.821702 12843 raft_consensus.cc:3037] T 3c978e0e91a348ecb03253b417e070a2 P 1463aabdbc9c46abbcff8fe150be4a5c [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:19.822769 12606 raft_consensus.cc:2454] T 3c978e0e91a348ecb03253b417e070a2 P 86cfc168c7df4b6ba49c45f12e9b9baa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77238a9fb1fd4965ac124fd5cdce98c9 in term 1.
I20260810 07:29:19.822984 12843 raft_consensus.cc:2454] T 3c978e0e91a348ecb03253b417e070a2 P 1463aabdbc9c46abbcff8fe150be4a5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77238a9fb1fd4965ac124fd5cdce98c9 in term 1.
I20260810 07:29:19.823102 12958 leader_election.cc:302] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77238a9fb1fd4965ac124fd5cdce98c9, 86cfc168c7df4b6ba49c45f12e9b9baa; no voters:
I20260810 07:29:19.823297 13189 raft_consensus.cc:2781] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:19.823552 13189 raft_consensus.cc:686] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9 [term 1 LEADER]: Becoming Leader. State: Replica: 77238a9fb1fd4965ac124fd5cdce98c9, State: Running, Role: LEADER
I20260810 07:29:19.823884 13189 consensus_queue.cc:237] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cfc168c7df4b6ba49c45f12e9b9baa" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 43859 } } peers { permanent_uuid: "77238a9fb1fd4965ac124fd5cdce98c9" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41275 } } peers { permanent_uuid: "1463aabdbc9c46abbcff8fe150be4a5c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 41753 } }
I20260810 07:29:19.827172 12367 catalog_manager.cc:5495] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77238a9fb1fd4965ac124fd5cdce98c9 (127.12.157.195). New cstate: current_term: 1 leader_uuid: "77238a9fb1fd4965ac124fd5cdce98c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86cfc168c7df4b6ba49c45f12e9b9baa" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 43859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77238a9fb1fd4965ac124fd5cdce98c9" member_type: VOTER last_known_addr { host: "127.12.157.195" port: 41275 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1463aabdbc9c46abbcff8fe150be4a5c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 41753 } health_report { overall_health: UNKNOWN } } }
I20260810 07:29:19.883508 13170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:19.880253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37598 (local address 127.12.157.195:41275)
0810 07:29:19.880410 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:19.880414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:19.880427 (+ 13us) server_negotiation.cc:407] Connection header received
0810 07:29:19.880510 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:19.880512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:19.880567 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:19.880658 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:29:19.880873 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.881397 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.882224 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:19.882427 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:19.882503 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:19.882768 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:29:19.883179 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:29:19.883268 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:19.883328 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 07:29:19.883375 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 07:29:19.889482 12843 raft_consensus.cc:1261] T 3c978e0e91a348ecb03253b417e070a2 P 1463aabdbc9c46abbcff8fe150be4a5c [term 1 FOLLOWER]: Refusing update from remote peer 77238a9fb1fd4965ac124fd5cdce98c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:29:19.889487 12606 raft_consensus.cc:1261] T 3c978e0e91a348ecb03253b417e070a2 P 86cfc168c7df4b6ba49c45f12e9b9baa [term 1 FOLLOWER]: Refusing update from remote peer 77238a9fb1fd4965ac124fd5cdce98c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:29:19.890204 13196 consensus_queue.cc:1035] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86cfc168c7df4b6ba49c45f12e9b9baa" member_type: VOTER last_known_addr { host: "127.12.157.193" port: 43859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:29:19.890404 13189 consensus_queue.cc:1035] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1463aabdbc9c46abbcff8fe150be4a5c" member_type: VOTER last_known_addr { host: "127.12.157.194" port: 41753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:29:19.898124 13199 mvcc.cc:205] Tried to move back new op lower bound from 7316877147699982336 to 7316877147441356800. Current Snapshot: MvccSnapshot[applied={T|T < 7316877147699982336}]
I20260810 07:29:19.898455 13204 mvcc.cc:205] Tried to move back new op lower bound from 7316877147699982336 to 7316877147441356800. Current Snapshot: MvccSnapshot[applied={T|T < 7316877147699982336}]
I20260810 07:29:19.899021 13203 mvcc.cc:205] Tried to move back new op lower bound from 7316877147699982336 to 7316877147441356800. Current Snapshot: MvccSnapshot[applied={T|T < 7316877147699982336}]
I20260810 07:29:19.907709 12361 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:29:19.907867 12361 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 07:29:19.910377 12361 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 51cde13322fe4bdea9f23b9d665ef718: Sending DeleteTablet for 3 replicas of tablet 3c978e0e91a348ecb03253b417e070a2
I20260810 07:29:19.911262 13039 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c978e0e91a348ecb03253b417e070a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:29:19 UTC) from {username='slave'} at 127.0.0.1:37592
I20260810 07:29:19.911312 12804 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c978e0e91a348ecb03253b417e070a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:29:19 UTC) from {username='slave'} at 127.0.0.1:50354
I20260810 07:29:19.911324 12566 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c978e0e91a348ecb03253b417e070a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 07:29:19 UTC) from {username='slave'} at 127.0.0.1:51784
I20260810 07:29:19.911981 13224 tablet_replica.cc:330] T 3c978e0e91a348ecb03253b417e070a2 P 86cfc168c7df4b6ba49c45f12e9b9baa: stopping tablet replica
I20260810 07:29:19.911996 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 12472
I20260810 07:29:19.918154 13222 tablet_replica.cc:330] T 3c978e0e91a348ecb03253b417e070a2 P 1463aabdbc9c46abbcff8fe150be4a5c: stopping tablet replica
I20260810 07:29:19.918628 13222 raft_consensus.cc:2229] T 3c978e0e91a348ecb03253b417e070a2 P 1463aabdbc9c46abbcff8fe150be4a5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 07:29:19.919101 13222 raft_consensus.cc:2258] T 3c978e0e91a348ecb03253b417e070a2 P 1463aabdbc9c46abbcff8fe150be4a5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:29:19.922005 13222 ts_tablet_manager.cc:1905] T 3c978e0e91a348ecb03253b417e070a2 P 1463aabdbc9c46abbcff8fe150be4a5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 07:29:19.924286 13222 ts_tablet_manager.cc:1918] T 3c978e0e91a348ecb03253b417e070a2 P 1463aabdbc9c46abbcff8fe150be4a5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:29:19.924418 13222 log.cc:1196] T 3c978e0e91a348ecb03253b417e070a2 P 1463aabdbc9c46abbcff8fe150be4a5c: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-1/wal/wals/3c978e0e91a348ecb03253b417e070a2
I20260810 07:29:19.924764 13222 ts_tablet_manager.cc:1939] T 3c978e0e91a348ecb03253b417e070a2 P 1463aabdbc9c46abbcff8fe150be4a5c: Deleting consensus metadata
I20260810 07:29:19.924741 13220 tablet_replica.cc:330] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9: stopping tablet replica
I20260810 07:29:19.925204 13220 raft_consensus.cc:2229] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9 [term 1 LEADER]: Raft consensus shutting down.
I20260810 07:29:19.925706 13220 raft_consensus.cc:2258] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 07:29:19.926170 12342 catalog_manager.cc:4841] TS 1463aabdbc9c46abbcff8fe150be4a5c (127.12.157.194:41753): tablet 3c978e0e91a348ecb03253b417e070a2 (table test-table [id=e65bc4bd06c94f1abcc4adaff76e4f9a]) successfully deleted
I20260810 07:29:19.927395 13220 ts_tablet_manager.cc:1905] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 07:29:19.929431 12340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.157.193:43859 (error 108)
I20260810 07:29:19.929839 13220 ts_tablet_manager.cc:1918] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 07:29:19.929960 13220 log.cc:1196] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9: Deleting WAL directory at /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786346694975122-12919-0/minicluster-data/ts-2/wal/wals/3c978e0e91a348ecb03253b417e070a2
I20260810 07:29:19.930313 13220 ts_tablet_manager.cc:1939] T 3c978e0e91a348ecb03253b417e070a2 P 77238a9fb1fd4965ac124fd5cdce98c9: Deleting consensus metadata
I20260810 07:29:19.930657 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 12691
I20260810 07:29:19.930721 13168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:19.930426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.193:43859 (local address 127.0.0.1:51794)
0810 07:29:19.930556 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:19.930632 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.193:43859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 07:29:19.935277 12341 catalog_manager.cc:4841] TS 77238a9fb1fd4965ac124fd5cdce98c9 (127.12.157.195:41275): tablet 3c978e0e91a348ecb03253b417e070a2 (table test-table [id=e65bc4bd06c94f1abcc4adaff76e4f9a]) successfully deleted
W20260810 07:29:19.935887 12340 catalog_manager.cc:4584] TS 86cfc168c7df4b6ba49c45f12e9b9baa (127.12.157.193:43859): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3c978e0e91a348ecb03253b417e070a2: Network error: Client connection negotiation failed: client connection to 127.12.157.193:43859: connect: Connection refused (error 111)
I20260810 07:29:19.945349 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 12931
I20260810 07:29:19.960687 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 12315
2026-08-10T07:29:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1375 ms)
[----------] 6 tests from EncryptionPolicyTest (11968 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:29:20 dist-test-slave-ffrd krb5kdc[13232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:29:20 dist-test-slave-ffrd krb5kdc[13232](info): set up 2 sockets
Aug 10 07:29:20 dist-test-slave-ffrd krb5kdc[13232](info): commencing operation
krb5kdc: starting...
W20260810 07:29:21.255872 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.262s user 0.006s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:29:21 dist-test-slave-ffrd krb5kdc[13232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:29:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 07:29:21.388100 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34651
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:21.517241 13247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:21.517645 13247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:21.517752 13247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:21.524051 13247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:21.524152 13247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:21.524189 13247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:21.524226 13247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:21.524259 13247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:21.530115 13247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34651
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:21.531328 13247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:21.532426 13247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:21.541508 13253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:21.541517 13256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:21.541587 13247 server_base.cc:1034] running on GCE node
W20260810 07:29:21.541504 13254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:21.542439 13247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:21.543659 13247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:21.544904 13247 hybrid_clock.cc:648] HybridClock initialized: now 1786346961544879 us; error 48 us; skew 500 ppm
Aug 10 07:29:21 dist-test-slave-ffrd krb5kdc[13232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:21.548121 13247 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 07:29:21.549437 13247 webserver.cc:460] Webserver started at http://127.0.0.1:35841/ using document root <none> and password file <none>
I20260810 07:29:21.550112 13247 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:21.550222 13247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:21.550487 13247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:21.552754 13247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "acbc452287894dd98d025b187171db12"
format_stamp: "Formatted at 2026-08-10 07:29:21 on dist-test-slave-ffrd"
server_key: "d73a99b80ab4704bd0a9d11b39a380ff"
server_key_iv: "352e78f816bb53f12fa8b32f8c976af8"
server_key_version: "encryptionkey@0"
I20260810 07:29:21.553432 13247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "acbc452287894dd98d025b187171db12"
format_stamp: "Formatted at 2026-08-10 07:29:21 on dist-test-slave-ffrd"
server_key: "d73a99b80ab4704bd0a9d11b39a380ff"
server_key_iv: "352e78f816bb53f12fa8b32f8c976af8"
server_key_version: "encryptionkey@0"
I20260810 07:29:21.557592 13247 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 07:29:21.560185 13264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:21.561638 13247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:29:21.561796 13247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "acbc452287894dd98d025b187171db12"
format_stamp: "Formatted at 2026-08-10 07:29:21 on dist-test-slave-ffrd"
server_key: "d73a99b80ab4704bd0a9d11b39a380ff"
server_key_iv: "352e78f816bb53f12fa8b32f8c976af8"
server_key_version: "encryptionkey@0"
I20260810 07:29:21.561925 13247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:21.591334 13247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:21.594779 13247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:21.595026 13247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:21.603195 13247 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34651
I20260810 07:29:21.603205 13340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34651 every 8 connection(s)
I20260810 07:29:21.604534 13247 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:21.607074 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 13247
I20260810 07:29:21.607256 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:29:21.607595 12919 external_mini_cluster.cc:1428] Setting key fd10b392209e5a61fa83fb311389aad5
I20260810 07:29:21.607630 13341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:21.615353 13341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acbc452287894dd98d025b187171db12: Bootstrap starting.
Aug 10 07:29:21 dist-test-slave-ffrd krb5kdc[13232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 07:29:21.618100 13341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acbc452287894dd98d025b187171db12: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:21.619009 13341 log.cc:824] T 00000000000000000000000000000000 P acbc452287894dd98d025b187171db12: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:21.621492 13341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acbc452287894dd98d025b187171db12: No bootstrap required, opened a new log
I20260810 07:29:21.624436 13348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:21.609237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40079 (local address 127.0.0.1:34651)
0810 07:29:21.609697 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:21.609718 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:21.609761 (+ 43us) server_negotiation.cc:407] Connection header received
0810 07:29:21.610619 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:21.610644 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:21.611093 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:21.611476 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:21.612195 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:21.613773 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:21.614662 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:21.615341 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:21.615512 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:21.618628 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:21.618635 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:21.618651 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:21.618682 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:21.620854 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:21.621610 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:21.621616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:21.621622 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:21.621764 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:21.622110 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:21.622113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:21.622115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:21.622614 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:21.622861 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:21.623140 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 07:29:21.623373 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":135,"threads_started":1}
I20260810 07:29:21.626190 13341 raft_consensus.cc:348] T 00000000000000000000000000000000 P acbc452287894dd98d025b187171db12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acbc452287894dd98d025b187171db12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34651 } }
I20260810 07:29:21.626641 13341 raft_consensus.cc:374] T 00000000000000000000000000000000 P acbc452287894dd98d025b187171db12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:21.626711 13341 raft_consensus.cc:729] T 00000000000000000000000000000000 P acbc452287894dd98d025b187171db12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acbc452287894dd98d025b187171db12, State: Initialized, Role: FOLLOWER
I20260810 07:29:21.627229 13341 consensus_queue.cc:260] T 00000000000000000000000000000000 P acbc452287894dd98d025b187171db12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acbc452287894dd98d025b187171db12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34651 } }
I20260810 07:29:21.627415 13341 raft_consensus.cc:388] T 00000000000000000000000000000000 P acbc452287894dd98d025b187171db12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:29:21.627486 13341 raft_consensus.cc:482] T 00000000000000000000000000000000 P acbc452287894dd98d025b187171db12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:29:21.627583 13341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acbc452287894dd98d025b187171db12 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:21.627981 12919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:29:21.628672 13341 raft_consensus.cc:504] T 00000000000000000000000000000000 P acbc452287894dd98d025b187171db12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acbc452287894dd98d025b187171db12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34651 } }
I20260810 07:29:21.629088 13341 leader_election.cc:302] T 00000000000000000000000000000000 P acbc452287894dd98d025b187171db12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acbc452287894dd98d025b187171db12; no voters:
I20260810 07:29:21.629460 13341 leader_election.cc:288] T 00000000000000000000000000000000 P acbc452287894dd98d025b187171db12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:29:21.629581 13351 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acbc452287894dd98d025b187171db12 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:21.629808 13351 raft_consensus.cc:686] T 00000000000000000000000000000000 P acbc452287894dd98d025b187171db12 [term 1 LEADER]: Becoming Leader. State: Replica: acbc452287894dd98d025b187171db12, State: Running, Role: LEADER
I20260810 07:29:21.630142 13351 consensus_queue.cc:237] T 00000000000000000000000000000000 P acbc452287894dd98d025b187171db12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acbc452287894dd98d025b187171db12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34651 } }
I20260810 07:29:21.630843 13341 sys_catalog.cc:564] T 00000000000000000000000000000000 P acbc452287894dd98d025b187171db12 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:21.631982 13357 sys_catalog.cc:455] T 00000000000000000000000000000000 P acbc452287894dd98d025b187171db12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acbc452287894dd98d025b187171db12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acbc452287894dd98d025b187171db12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34651 } } }
I20260810 07:29:21.632195 13357 sys_catalog.cc:458] T 00000000000000000000000000000000 P acbc452287894dd98d025b187171db12 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:21.632601 13361 sys_catalog.cc:455] T 00000000000000000000000000000000 P acbc452287894dd98d025b187171db12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acbc452287894dd98d025b187171db12. Latest consensus state: current_term: 1 leader_uuid: "acbc452287894dd98d025b187171db12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acbc452287894dd98d025b187171db12" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34651 } } }
I20260810 07:29:21.632805 13361 sys_catalog.cc:458] T 00000000000000000000000000000000 P acbc452287894dd98d025b187171db12 [sys.catalog]: This master's current role is: LEADER
W20260810 07:29:21.636122 13392 catalog_manager.cc:1558] T 00000000000000000000000000000000 P acbc452287894dd98d025b187171db12: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:29:21.636216 13392 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:29:21.636296 13378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:21.637316 13378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:21.640105 13348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:21.630492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34461 (local address 127.0.0.1:34651)
0810 07:29:21.630616 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:21.630621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:21.630875 (+ 254us) server_negotiation.cc:407] Connection header received
0810 07:29:21.631046 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:21.631050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:21.631098 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:21.631192 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:21.632079 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:21.632864 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:21.634078 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:21.634396 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:21.634515 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:21.636348 (+ 1833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:21.636352 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:21.636355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:21.636379 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:21.638092 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:21.638670 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:21.638674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:21.638676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:21.638764 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:21.639157 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:21.639160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:21.639162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:21.639373 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:21.639539 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:21.639829 (+ 290us) server_negotiation.cc:299] Negotiation successful
0810 07:29:21.639926 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 07:29:21.644256 13378 catalog_manager.cc:1347] Generated new cluster ID: ca9b5e27d27b40449041d5da0ab7fc7e
I20260810 07:29:21.644340 13378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:21.661065 13378 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:21.662539 13378 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:21.675439 13378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acbc452287894dd98d025b187171db12: Generated new TSK 0
I20260810 07:29:21.676177 13378 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:29:21.688208 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 13247
W20260810 07:29:21.701491 13227 connection.cc:535] client connection to 127.0.0.1:34651 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34651): Connection reset by peer (error 104)
2026-08-10T07:29:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1722 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:29:21 dist-test-slave-ffrd krb5kdc[13403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:29:21 dist-test-slave-ffrd krb5kdc[13403](info): set up 2 sockets
Aug 10 07:29:21 dist-test-slave-ffrd krb5kdc[13403](info): commencing operation
krb5kdc: starting...
W20260810 07:29:22.968214 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.252s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:29:23 dist-test-slave-ffrd krb5kdc[13403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:29:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 07:29:23.105583 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33313
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:23.221801 13422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:23.222110 13422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:23.222185 13422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:23.226145 13422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:23.226239 13422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:23.226277 13422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:23.226342 13422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:23.226382 13422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:23.231525 13422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33313
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:23.232620 13422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:23.233716 13422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:23.242394 13427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:23.242483 13428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:23.242512 13431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:23.242656 13422 server_base.cc:1034] running on GCE node
I20260810 07:29:23.243393 13422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:23.244587 13422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:23.245792 13422 hybrid_clock.cc:648] HybridClock initialized: now 1786346963245759 us; error 60 us; skew 500 ppm
Aug 10 07:29:23 dist-test-slave-ffrd krb5kdc[13403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346963, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:23.248687 13422 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 07:29:23.249678 13422 webserver.cc:460] Webserver started at http://127.0.0.1:41287/ using document root <none> and password file <none>
I20260810 07:29:23.250278 13422 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:23.250379 13422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:23.250633 13422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:23.252811 13422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "7ffb318703614e58afd0b0d8b15dd4d4"
format_stamp: "Formatted at 2026-08-10 07:29:23 on dist-test-slave-ffrd"
server_key: "da04d556825c550ba24597a3334fbb81"
server_key_iv: "0b68595ba04a40cbb0143e2ae905ff34"
server_key_version: "encryptionkey@0"
I20260810 07:29:23.253491 13422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "7ffb318703614e58afd0b0d8b15dd4d4"
format_stamp: "Formatted at 2026-08-10 07:29:23 on dist-test-slave-ffrd"
server_key: "da04d556825c550ba24597a3334fbb81"
server_key_iv: "0b68595ba04a40cbb0143e2ae905ff34"
server_key_version: "encryptionkey@0"
I20260810 07:29:23.257411 13422 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 07:29:23.259855 13445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:23.261111 13422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:29:23.261255 13422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "7ffb318703614e58afd0b0d8b15dd4d4"
format_stamp: "Formatted at 2026-08-10 07:29:23 on dist-test-slave-ffrd"
server_key: "da04d556825c550ba24597a3334fbb81"
server_key_iv: "0b68595ba04a40cbb0143e2ae905ff34"
server_key_version: "encryptionkey@0"
I20260810 07:29:23.261375 13422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:23.289328 13422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:23.293576 13422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:23.293825 13422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:23.302626 13422 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33313
I20260810 07:29:23.302647 13524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33313 every 8 connection(s)
I20260810 07:29:23.303737 13422 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:23.306725 13525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:23.312383 13525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ffb318703614e58afd0b0d8b15dd4d4: Bootstrap starting.
I20260810 07:29:23.313418 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 13422
I20260810 07:29:23.313539 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:29:23.313886 12919 external_mini_cluster.cc:1428] Setting key f02eff7ca8767f21886fbd89196591ab
I20260810 07:29:23.315207 13525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ffb318703614e58afd0b0d8b15dd4d4: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:23.316030 13525 log.cc:824] T 00000000000000000000000000000000 P 7ffb318703614e58afd0b0d8b15dd4d4: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:23.318075 13525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ffb318703614e58afd0b0d8b15dd4d4: No bootstrap required, opened a new log
I20260810 07:29:23.321813 13525 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ffb318703614e58afd0b0d8b15dd4d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ffb318703614e58afd0b0d8b15dd4d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33313 } }
I20260810 07:29:23.322036 13525 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ffb318703614e58afd0b0d8b15dd4d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:23.322069 13525 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ffb318703614e58afd0b0d8b15dd4d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ffb318703614e58afd0b0d8b15dd4d4, State: Initialized, Role: FOLLOWER
I20260810 07:29:23.322595 13525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ffb318703614e58afd0b0d8b15dd4d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ffb318703614e58afd0b0d8b15dd4d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33313 } }
I20260810 07:29:23.322772 13525 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ffb318703614e58afd0b0d8b15dd4d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:29:23.322850 13525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ffb318703614e58afd0b0d8b15dd4d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:29:23.322991 13525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ffb318703614e58afd0b0d8b15dd4d4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:23.323668 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:23.315096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41371 (local address 127.0.0.1:33313)
0810 07:29:23.315569 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:23.315584 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:23.315622 (+ 38us) server_negotiation.cc:407] Connection header received
0810 07:29:23.316480 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:23.316507 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:23.317015 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:23.317364 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:23.317803 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:23.318863 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:23.319816 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:23.320320 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:23.320645 (+ 325us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:23.321693 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:23.321698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:23.321711 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:23.322212 (+ 501us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:23.322275 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:23.322302 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:23.322450 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:23.322650 (+ 200us) server_negotiation.cc:299] Negotiation successful
0810 07:29:23.322798 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":150,"threads_started":1}
I20260810 07:29:23.324002 13525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ffb318703614e58afd0b0d8b15dd4d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ffb318703614e58afd0b0d8b15dd4d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33313 } }
I20260810 07:29:23.324355 13525 leader_election.cc:302] T 00000000000000000000000000000000 P 7ffb318703614e58afd0b0d8b15dd4d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ffb318703614e58afd0b0d8b15dd4d4; no voters:
I20260810 07:29:23.324707 13525 leader_election.cc:288] T 00000000000000000000000000000000 P 7ffb318703614e58afd0b0d8b15dd4d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:29:23.324832 13531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ffb318703614e58afd0b0d8b15dd4d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:23.325047 13531 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ffb318703614e58afd0b0d8b15dd4d4 [term 1 LEADER]: Becoming Leader. State: Replica: 7ffb318703614e58afd0b0d8b15dd4d4, State: Running, Role: LEADER
I20260810 07:29:23.325340 13531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ffb318703614e58afd0b0d8b15dd4d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ffb318703614e58afd0b0d8b15dd4d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33313 } }
I20260810 07:29:23.326033 13525 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ffb318703614e58afd0b0d8b15dd4d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:23.326540 13531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ffb318703614e58afd0b0d8b15dd4d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ffb318703614e58afd0b0d8b15dd4d4. Latest consensus state: current_term: 1 leader_uuid: "7ffb318703614e58afd0b0d8b15dd4d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ffb318703614e58afd0b0d8b15dd4d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33313 } } }
I20260810 07:29:23.326649 13531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ffb318703614e58afd0b0d8b15dd4d4 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:23.327065 13540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:23.327091 12919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:29:23.327005 13534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ffb318703614e58afd0b0d8b15dd4d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ffb318703614e58afd0b0d8b15dd4d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ffb318703614e58afd0b0d8b15dd4d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33313 } } }
I20260810 07:29:23.327144 13534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ffb318703614e58afd0b0d8b15dd4d4 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:23.330415 13540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:23.336866 13540 catalog_manager.cc:1347] Generated new cluster ID: 012a1d6537b54b06bca7ef070afee74a
I20260810 07:29:23.336990 13540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:23.342017 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:23.329891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59109 (local address 127.0.0.1:33313)
0810 07:29:23.333094 (+ 3203us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:23.333099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:23.333114 (+ 15us) server_negotiation.cc:407] Connection header received
0810 07:29:23.333153 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:23.333156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:23.333201 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:23.333359 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:23.333720 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:23.334475 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:23.335496 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:23.335761 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:23.335876 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:23.341430 (+ 5554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:23.341434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:23.341437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:23.341455 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:23.341490 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:23.341504 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:23.341588 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:23.341825 (+ 237us) server_negotiation.cc:299] Negotiation successful
0810 07:29:23.341866 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3119}
I20260810 07:29:23.354391 13540 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:23.355333 13540 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:23.367303 13540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ffb318703614e58afd0b0d8b15dd4d4: Generated new TSK 0
I20260810 07:29:23.368045 13540 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:29:23.378986 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 13422
W20260810 07:29:23.397326 13400 connection.cc:535] client connection to 127.0.0.1:33313 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33313): Connection reset by peer (error 104)
2026-08-10T07:29:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1695 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:29:23 dist-test-slave-ffrd krb5kdc[13580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:29:23 dist-test-slave-ffrd krb5kdc[13580](info): set up 2 sockets
Aug 10 07:29:23 dist-test-slave-ffrd krb5kdc[13580](info): commencing operation
krb5kdc: starting...
W20260810 07:29:24.704635 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.293s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:29:24 dist-test-slave-ffrd krb5kdc[13580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:29:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 07:29:24.843997 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43385
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:24.959425 13603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:24.959718 13603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:24.959796 13603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:24.963831 13603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:24.963930 13603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:24.963967 13603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:24.964023 13603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:24.964062 13603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:24.969326 13603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43385
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:24.970392 13603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:24.971568 13603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:24.980383 13619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:24.980394 13615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:24.980414 13614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:24.980672 13603 server_base.cc:1034] running on GCE node
I20260810 07:29:24.981350 13603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:24.982460 13603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:24.983661 13603 hybrid_clock.cc:648] HybridClock initialized: now 1786346964983644 us; error 31 us; skew 500 ppm
Aug 10 07:29:24 dist-test-slave-ffrd krb5kdc[13580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:24.991931 13603 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 07:29:24.993031 13603 webserver.cc:460] Webserver started at http://127.0.0.1:42135/ using document root <none> and password file <none>
I20260810 07:29:24.993603 13603 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:24.993665 13603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:24.993844 13603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:24.995939 13603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "0d923531f6cc4fe38e6cdac0065c9cb9"
format_stamp: "Formatted at 2026-08-10 07:29:24 on dist-test-slave-ffrd"
server_key: "4a1a7214aac9c0d2771a9a4cfccb66cf"
server_key_iv: "0c85404ef82cc4445f7b7c7c5408faa0"
server_key_version: "encryptionkey@0"
I20260810 07:29:24.996500 13603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "0d923531f6cc4fe38e6cdac0065c9cb9"
format_stamp: "Formatted at 2026-08-10 07:29:24 on dist-test-slave-ffrd"
server_key: "4a1a7214aac9c0d2771a9a4cfccb66cf"
server_key_iv: "0c85404ef82cc4445f7b7c7c5408faa0"
server_key_version: "encryptionkey@0"
I20260810 07:29:25.000509 13603 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 07:29:25.003146 13632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:25.004527 13603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 07:29:25.004652 13603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "0d923531f6cc4fe38e6cdac0065c9cb9"
format_stamp: "Formatted at 2026-08-10 07:29:24 on dist-test-slave-ffrd"
server_key: "4a1a7214aac9c0d2771a9a4cfccb66cf"
server_key_iv: "0c85404ef82cc4445f7b7c7c5408faa0"
server_key_version: "encryptionkey@0"
I20260810 07:29:25.004753 13603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:25.009583 13603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:25.013020 13603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:25.013187 13603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:25.021066 13603 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43385
I20260810 07:29:25.021093 13727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43385 every 8 connection(s)
I20260810 07:29:25.022116 13603 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:25.025267 13728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:25.030812 13728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d923531f6cc4fe38e6cdac0065c9cb9: Bootstrap starting.
I20260810 07:29:25.032087 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 13603
I20260810 07:29:25.032235 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:29:25.032527 12919 external_mini_cluster.cc:1428] Setting key 6030583e80e3eaf85d30b066d6e14ce5
I20260810 07:29:25.033759 13728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d923531f6cc4fe38e6cdac0065c9cb9: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:25.034539 13728 log.cc:824] T 00000000000000000000000000000000 P 0d923531f6cc4fe38e6cdac0065c9cb9: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:25.036932 13728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d923531f6cc4fe38e6cdac0065c9cb9: No bootstrap required, opened a new log
I20260810 07:29:25.038175 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:25.033815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55015 (local address 127.0.0.1:43385)
0810 07:29:25.034326 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:25.034366 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:25.034402 (+ 36us) server_negotiation.cc:407] Connection header received
0810 07:29:25.035333 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:25.035378 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:25.035651 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:25.035936 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:25.036856 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:25.036863 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:25.036876 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:25.036887 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:25.037010 (+ 123us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:25.037036 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:25.037174 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:25.037290 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 07:29:25.037480 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":162,"threads_started":1}
I20260810 07:29:25.040925 12919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:29:25.040805 13728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d923531f6cc4fe38e6cdac0065c9cb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d923531f6cc4fe38e6cdac0065c9cb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43385 } }
I20260810 07:29:25.041131 13728 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d923531f6cc4fe38e6cdac0065c9cb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:25.041195 13728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d923531f6cc4fe38e6cdac0065c9cb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d923531f6cc4fe38e6cdac0065c9cb9, State: Initialized, Role: FOLLOWER
I20260810 07:29:25.041702 13728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d923531f6cc4fe38e6cdac0065c9cb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d923531f6cc4fe38e6cdac0065c9cb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43385 } }
I20260810 07:29:25.041870 13728 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d923531f6cc4fe38e6cdac0065c9cb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:29:25.041941 13728 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d923531f6cc4fe38e6cdac0065c9cb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:29:25.042034 13728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d923531f6cc4fe38e6cdac0065c9cb9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:25.043337 13728 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d923531f6cc4fe38e6cdac0065c9cb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d923531f6cc4fe38e6cdac0065c9cb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43385 } }
I20260810 07:29:25.043735 13728 leader_election.cc:302] T 00000000000000000000000000000000 P 0d923531f6cc4fe38e6cdac0065c9cb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d923531f6cc4fe38e6cdac0065c9cb9; no voters:
I20260810 07:29:25.044116 13728 leader_election.cc:288] T 00000000000000000000000000000000 P 0d923531f6cc4fe38e6cdac0065c9cb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:29:25.044214 13740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d923531f6cc4fe38e6cdac0065c9cb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:25.044811 13740 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d923531f6cc4fe38e6cdac0065c9cb9 [term 1 LEADER]: Becoming Leader. State: Replica: 0d923531f6cc4fe38e6cdac0065c9cb9, State: Running, Role: LEADER
I20260810 07:29:25.045248 13740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d923531f6cc4fe38e6cdac0065c9cb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d923531f6cc4fe38e6cdac0065c9cb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43385 } }
I20260810 07:29:25.045388 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:25.043400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40271 (local address 127.0.0.1:43385)
0810 07:29:25.043541 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:25.043545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:25.043783 (+ 238us) server_negotiation.cc:407] Connection header received
0810 07:29:25.043932 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:25.043935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:25.043982 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:25.044050 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:25.044886 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:25.044892 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:25.044896 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:25.044910 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:25.044964 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:25.044980 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:25.045041 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:25.045220 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 07:29:25.045261 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 07:29:25.045557 13728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d923531f6cc4fe38e6cdac0065c9cb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:25.047036 13741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d923531f6cc4fe38e6cdac0065c9cb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d923531f6cc4fe38e6cdac0065c9cb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d923531f6cc4fe38e6cdac0065c9cb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43385 } } }
I20260810 07:29:25.047183 13741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d923531f6cc4fe38e6cdac0065c9cb9 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:25.047603 13756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:25.047566 13742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d923531f6cc4fe38e6cdac0065c9cb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d923531f6cc4fe38e6cdac0065c9cb9. Latest consensus state: current_term: 1 leader_uuid: "0d923531f6cc4fe38e6cdac0065c9cb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d923531f6cc4fe38e6cdac0065c9cb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43385 } } }
I20260810 07:29:25.048233 13742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d923531f6cc4fe38e6cdac0065c9cb9 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:25.050544 13756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:25.056279 13756 catalog_manager.cc:1347] Generated new cluster ID: 23dbedb010fd464bbae6fb087747e5a6
I20260810 07:29:25.056360 13756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:25.068456 13756 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:25.068847 13756 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:25.075045 13756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d923531f6cc4fe38e6cdac0065c9cb9: Generated new TSK 0
I20260810 07:29:25.075692 13756 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:29:25.080482 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 13603
2026-08-10T07:29:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1694 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:29:25 dist-test-slave-ffrd krb5kdc[13774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:29:25 dist-test-slave-ffrd krb5kdc[13774](info): set up 2 sockets
Aug 10 07:29:25 dist-test-slave-ffrd krb5kdc[13774](info): commencing operation
krb5kdc: starting...
W20260810 07:29:26.362666 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.256s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:29:26 dist-test-slave-ffrd krb5kdc[13774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:29:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 07:29:26.499480 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41207
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:26.615928 13790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:26.616206 13790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:26.616248 13790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:26.620407 13790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:26.620482 13790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:26.620503 13790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:26.620520 13790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:26.620536 13790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:26.626250 13790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:26.627317 13790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:26.628328 13790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:26.637394 13808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:26.637488 13805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:26.637569 13802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:26.637674 13790 server_base.cc:1034] running on GCE node
I20260810 07:29:26.638625 13790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:26.640082 13790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:26.641314 13790 hybrid_clock.cc:648] HybridClock initialized: now 1786346966641284 us; error 45 us; skew 500 ppm
Aug 10 07:29:26 dist-test-slave-ffrd krb5kdc[13774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:26.644328 13790 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 07:29:26.645406 13790 webserver.cc:460] Webserver started at http://127.0.0.1:37053/ using document root <none> and password file <none>
I20260810 07:29:26.645955 13790 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:26.646015 13790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:26.646188 13790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:26.648438 13790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "e75ef0a36cf542698a3d041da60d593d"
format_stamp: "Formatted at 2026-08-10 07:29:26 on dist-test-slave-ffrd"
server_key: "98133f2d7f906ade0a2179628ddd9c9f"
server_key_iv: "5fd87fd424d358f1b39d811736ab645b"
server_key_version: "encryptionkey@0"
I20260810 07:29:26.649044 13790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "e75ef0a36cf542698a3d041da60d593d"
format_stamp: "Formatted at 2026-08-10 07:29:26 on dist-test-slave-ffrd"
server_key: "98133f2d7f906ade0a2179628ddd9c9f"
server_key_iv: "5fd87fd424d358f1b39d811736ab645b"
server_key_version: "encryptionkey@0"
I20260810 07:29:26.652913 13790 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 07:29:26.655534 13828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:26.657073 13790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:29:26.657210 13790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "e75ef0a36cf542698a3d041da60d593d"
format_stamp: "Formatted at 2026-08-10 07:29:26 on dist-test-slave-ffrd"
server_key: "98133f2d7f906ade0a2179628ddd9c9f"
server_key_iv: "5fd87fd424d358f1b39d811736ab645b"
server_key_version: "encryptionkey@0"
I20260810 07:29:26.657327 13790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:26.669337 13790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:26.672852 13790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:26.673135 13790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:26.680836 13790 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41207
I20260810 07:29:26.680866 13913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41207 every 8 connection(s)
I20260810 07:29:26.681874 13790 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:26.684792 13914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:26.687395 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 13790
I20260810 07:29:26.687515 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:29:26.687789 12919 external_mini_cluster.cc:1428] Setting key b239150755ba40f4200b5348a7f7b6b5
I20260810 07:29:26.690850 13914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e75ef0a36cf542698a3d041da60d593d: Bootstrap starting.
I20260810 07:29:26.693612 13914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e75ef0a36cf542698a3d041da60d593d: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:26.694510 13914 log.cc:824] T 00000000000000000000000000000000 P e75ef0a36cf542698a3d041da60d593d: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:26.697054 13914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e75ef0a36cf542698a3d041da60d593d: No bootstrap required, opened a new log
Aug 10 07:29:26 dist-test-slave-ffrd krb5kdc[13774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 07:29:26.700060 13914 raft_consensus.cc:348] T 00000000000000000000000000000000 P e75ef0a36cf542698a3d041da60d593d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75ef0a36cf542698a3d041da60d593d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41207 } }
I20260810 07:29:26.700310 13914 raft_consensus.cc:374] T 00000000000000000000000000000000 P e75ef0a36cf542698a3d041da60d593d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:26.700410 13914 raft_consensus.cc:729] T 00000000000000000000000000000000 P e75ef0a36cf542698a3d041da60d593d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e75ef0a36cf542698a3d041da60d593d, State: Initialized, Role: FOLLOWER
I20260810 07:29:26.701049 13914 consensus_queue.cc:260] T 00000000000000000000000000000000 P e75ef0a36cf542698a3d041da60d593d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75ef0a36cf542698a3d041da60d593d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41207 } }
I20260810 07:29:26.701241 13914 raft_consensus.cc:388] T 00000000000000000000000000000000 P e75ef0a36cf542698a3d041da60d593d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:29:26.701323 13914 raft_consensus.cc:482] T 00000000000000000000000000000000 P e75ef0a36cf542698a3d041da60d593d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:29:26.701476 13914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e75ef0a36cf542698a3d041da60d593d [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:26.702869 13914 raft_consensus.cc:504] T 00000000000000000000000000000000 P e75ef0a36cf542698a3d041da60d593d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75ef0a36cf542698a3d041da60d593d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41207 } }
I20260810 07:29:26.703279 13914 leader_election.cc:302] T 00000000000000000000000000000000 P e75ef0a36cf542698a3d041da60d593d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e75ef0a36cf542698a3d041da60d593d; no voters:
I20260810 07:29:26.703632 13914 leader_election.cc:288] T 00000000000000000000000000000000 P e75ef0a36cf542698a3d041da60d593d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:29:26.703768 13925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e75ef0a36cf542698a3d041da60d593d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:26.703922 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:26.689293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53905 (local address 127.0.0.1:41207)
0810 07:29:26.689756 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:26.689772 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:26.689809 (+ 37us) server_negotiation.cc:407] Connection header received
0810 07:29:26.690441 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:26.690458 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:26.690794 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:26.691123 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:26.692049 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:26.693124 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:26.694326 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:26.694844 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:26.695039 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:26.697980 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:26.697986 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:26.698000 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:26.698027 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:26.700303 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:26.700865 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:26.700870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:26.700874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:26.701058 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:26.701410 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:26.701414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:26.701416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:26.701981 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:26.702210 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:26.702531 (+ 321us) server_negotiation.cc:299] Negotiation successful
0810 07:29:26.702887 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":124,"threads_started":1}
I20260810 07:29:26.704071 13925 raft_consensus.cc:686] T 00000000000000000000000000000000 P e75ef0a36cf542698a3d041da60d593d [term 1 LEADER]: Becoming Leader. State: Replica: e75ef0a36cf542698a3d041da60d593d, State: Running, Role: LEADER
I20260810 07:29:26.704411 13925 consensus_queue.cc:237] T 00000000000000000000000000000000 P e75ef0a36cf542698a3d041da60d593d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75ef0a36cf542698a3d041da60d593d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41207 } }
I20260810 07:29:26.704932 13914 sys_catalog.cc:564] T 00000000000000000000000000000000 P e75ef0a36cf542698a3d041da60d593d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:26.705729 13928 sys_catalog.cc:455] T 00000000000000000000000000000000 P e75ef0a36cf542698a3d041da60d593d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e75ef0a36cf542698a3d041da60d593d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75ef0a36cf542698a3d041da60d593d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41207 } } }
I20260810 07:29:26.705785 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P e75ef0a36cf542698a3d041da60d593d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e75ef0a36cf542698a3d041da60d593d. Latest consensus state: current_term: 1 leader_uuid: "e75ef0a36cf542698a3d041da60d593d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75ef0a36cf542698a3d041da60d593d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41207 } } }
I20260810 07:29:26.705875 13928 sys_catalog.cc:458] T 00000000000000000000000000000000 P e75ef0a36cf542698a3d041da60d593d [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:26.705904 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P e75ef0a36cf542698a3d041da60d593d [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:26.707304 12919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260810 07:29:26.710820 13950 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e75ef0a36cf542698a3d041da60d593d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:29:26.710916 13950 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:29:26.711019 13941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:26.711836 13941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:26.719224 13941 catalog_manager.cc:1347] Generated new cluster ID: 6b071093e8b74b93a7f4a910c38a5120
I20260810 07:29:26.719326 13941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:26.719379 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:26.709872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37851 (local address 127.0.0.1:41207)
0810 07:29:26.709981 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:26.709985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:26.710230 (+ 245us) server_negotiation.cc:407] Connection header received
0810 07:29:26.710352 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:26.710355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:26.710402 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:26.710493 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:26.711328 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:26.711902 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:26.713047 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:26.713285 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:26.713373 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:26.715073 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:26.715077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:26.715080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:26.715107 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:26.716840 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:26.717797 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:26.717801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:26.717803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:26.717900 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:26.718229 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:26.718233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:26.718235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:26.718478 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:26.718591 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:26.718970 (+ 379us) server_negotiation.cc:299] Negotiation successful
0810 07:29:26.719105 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 07:29:26.763301 13941 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:26.764581 13941 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:26.772557 13941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e75ef0a36cf542698a3d041da60d593d: Generated new TSK 0
I20260810 07:29:26.773478 13941 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:29:26.783492 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 13790
2026-08-10T07:29:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1715 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:29:26 dist-test-slave-ffrd krb5kdc[13971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:29:26 dist-test-slave-ffrd krb5kdc[13971](info): set up 2 sockets
Aug 10 07:29:26 dist-test-slave-ffrd krb5kdc[13971](info): commencing operation
krb5kdc: starting...
W20260810 07:29:28.078749 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.256s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:29:28 dist-test-slave-ffrd krb5kdc[13971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:29:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 07:29:28.212600 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43167
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:28.345886 13988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:28.346189 13988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:28.346264 13988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:28.350179 13988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:28.350275 13988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:28.350315 13988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:28.350374 13988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:28.350415 13988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:28.355556 13988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:28.356604 13988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:28.357632 13988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:28.365980 14000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:28.365995 14004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:28.366000 14001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:28.366266 13988 server_base.cc:1034] running on GCE node
I20260810 07:29:28.366915 13988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:28.367990 13988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:28.369181 13988 hybrid_clock.cc:648] HybridClock initialized: now 1786346968369169 us; error 30 us; skew 500 ppm
Aug 10 07:29:28 dist-test-slave-ffrd krb5kdc[13971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:28.372058 13988 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 07:29:28.373008 13988 webserver.cc:460] Webserver started at http://127.0.0.1:37601/ using document root <none> and password file <none>
I20260810 07:29:28.373566 13988 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:28.373659 13988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:28.373908 13988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:28.375963 13988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "ea24e8d5213540b0bb42d9cf4e1a048c"
format_stamp: "Formatted at 2026-08-10 07:29:28 on dist-test-slave-ffrd"
server_key: "9ed3359739fcabe36e33f139c8ce184a"
server_key_iv: "7cdbb424be3ad722f4d3037f44ce5ce1"
server_key_version: "encryptionkey@0"
I20260810 07:29:28.376531 13988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "ea24e8d5213540b0bb42d9cf4e1a048c"
format_stamp: "Formatted at 2026-08-10 07:29:28 on dist-test-slave-ffrd"
server_key: "9ed3359739fcabe36e33f139c8ce184a"
server_key_iv: "7cdbb424be3ad722f4d3037f44ce5ce1"
server_key_version: "encryptionkey@0"
I20260810 07:29:28.380260 13988 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 07:29:28.382755 14023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:28.383937 13988 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 07:29:28.384080 13988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "ea24e8d5213540b0bb42d9cf4e1a048c"
format_stamp: "Formatted at 2026-08-10 07:29:28 on dist-test-slave-ffrd"
server_key: "9ed3359739fcabe36e33f139c8ce184a"
server_key_iv: "7cdbb424be3ad722f4d3037f44ce5ce1"
server_key_version: "encryptionkey@0"
I20260810 07:29:28.384193 13988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:28.416677 13988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:28.419955 13988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:28.420217 13988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:28.428188 13988 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43167
I20260810 07:29:28.428213 14104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43167 every 8 connection(s)
I20260810 07:29:28.429266 13988 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:28.430557 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 13988
I20260810 07:29:28.430668 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:29:28.430984 12919 external_mini_cluster.cc:1428] Setting key b4f91fbd13d681c94419db13e2e43260
I20260810 07:29:28.432826 14105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:28.439025 14105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea24e8d5213540b0bb42d9cf4e1a048c: Bootstrap starting.
I20260810 07:29:28.440387 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:28.432451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41693 (local address 127.0.0.1:43167)
0810 07:29:28.433000 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:28.433026 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:28.433066 (+ 40us) server_negotiation.cc:407] Connection header received
0810 07:29:28.433793 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:28.433813 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:28.434157 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:28.434530 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:28.434949 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:28.435947 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:28.436903 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:28.437458 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:28.437641 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:28.438619 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:28.438624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:28.438637 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:28.439038 (+ 401us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:28.439124 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:28.439160 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:28.439289 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:28.439445 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 07:29:28.439674 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":144,"threads_started":1}
I20260810 07:29:28.441943 14105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea24e8d5213540b0bb42d9cf4e1a048c: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:28.442895 14105 log.cc:824] T 00000000000000000000000000000000 P ea24e8d5213540b0bb42d9cf4e1a048c: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:28.443723 12919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:29:28.445266 14105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea24e8d5213540b0bb42d9cf4e1a048c: No bootstrap required, opened a new log
I20260810 07:29:28.448992 14105 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea24e8d5213540b0bb42d9cf4e1a048c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea24e8d5213540b0bb42d9cf4e1a048c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43167 } }
I20260810 07:29:28.449306 14105 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea24e8d5213540b0bb42d9cf4e1a048c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:28.449378 14105 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea24e8d5213540b0bb42d9cf4e1a048c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea24e8d5213540b0bb42d9cf4e1a048c, State: Initialized, Role: FOLLOWER
I20260810 07:29:28.449896 14105 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea24e8d5213540b0bb42d9cf4e1a048c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea24e8d5213540b0bb42d9cf4e1a048c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43167 } }
I20260810 07:29:28.450066 14105 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea24e8d5213540b0bb42d9cf4e1a048c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:29:28.450150 14105 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea24e8d5213540b0bb42d9cf4e1a048c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:29:28.450271 14105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea24e8d5213540b0bb42d9cf4e1a048c [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:28.450448 14111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:28.446071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34335 (local address 127.0.0.1:43167)
0810 07:29:28.446205 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:28.446211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:28.446456 (+ 245us) server_negotiation.cc:407] Connection header received
0810 07:29:28.446609 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:28.446612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:28.446664 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:28.446746 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:28.447015 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:28.447552 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:28.448368 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:28.448622 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:28.448737 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:28.449716 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:28.449720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:28.449725 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:28.449741 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:28.449772 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:28.449783 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:28.449868 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:28.450185 (+ 317us) server_negotiation.cc:299] Negotiation successful
0810 07:29:28.450246 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 07:29:28.451613 14105 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea24e8d5213540b0bb42d9cf4e1a048c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea24e8d5213540b0bb42d9cf4e1a048c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43167 } }
I20260810 07:29:28.451982 14105 leader_election.cc:302] T 00000000000000000000000000000000 P ea24e8d5213540b0bb42d9cf4e1a048c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea24e8d5213540b0bb42d9cf4e1a048c; no voters:
I20260810 07:29:28.452317 14105 leader_election.cc:288] T 00000000000000000000000000000000 P ea24e8d5213540b0bb42d9cf4e1a048c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:29:28.452482 14121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea24e8d5213540b0bb42d9cf4e1a048c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:28.452745 14121 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea24e8d5213540b0bb42d9cf4e1a048c [term 1 LEADER]: Becoming Leader. State: Replica: ea24e8d5213540b0bb42d9cf4e1a048c, State: Running, Role: LEADER
I20260810 07:29:28.453148 14121 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea24e8d5213540b0bb42d9cf4e1a048c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea24e8d5213540b0bb42d9cf4e1a048c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43167 } }
I20260810 07:29:28.453718 14105 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea24e8d5213540b0bb42d9cf4e1a048c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:28.455016 14123 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea24e8d5213540b0bb42d9cf4e1a048c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea24e8d5213540b0bb42d9cf4e1a048c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea24e8d5213540b0bb42d9cf4e1a048c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43167 } } }
I20260810 07:29:28.455143 14123 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea24e8d5213540b0bb42d9cf4e1a048c [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:28.455495 14124 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea24e8d5213540b0bb42d9cf4e1a048c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea24e8d5213540b0bb42d9cf4e1a048c. Latest consensus state: current_term: 1 leader_uuid: "ea24e8d5213540b0bb42d9cf4e1a048c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea24e8d5213540b0bb42d9cf4e1a048c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43167 } } }
I20260810 07:29:28.455565 14142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:28.455628 14124 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea24e8d5213540b0bb42d9cf4e1a048c [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:28.458741 14142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:28.464299 14142 catalog_manager.cc:1347] Generated new cluster ID: c9bd149cfd5340049ca1d8d7d1cbf339
I20260810 07:29:28.464427 14142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:28.472618 14142 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:28.473604 14142 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:28.482930 14142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea24e8d5213540b0bb42d9cf4e1a048c: Generated new TSK 0
I20260810 07:29:28.483839 14142 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:29:28.493602 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 13988
2026-08-10T07:29:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1698 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:29:28 dist-test-slave-ffrd krb5kdc[14158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:29:28 dist-test-slave-ffrd krb5kdc[14158](info): set up 2 sockets
Aug 10 07:29:28 dist-test-slave-ffrd krb5kdc[14158](info): commencing operation
krb5kdc: starting...
W20260810 07:29:29.760805 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.240s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:29:29 dist-test-slave-ffrd krb5kdc[14158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346969, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:29:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 07:29:29.897253 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36687
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:30.015092 14175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:30.015532 14175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:30.015779 14175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:30.020177 14175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:30.020257 14175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:30.020277 14175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:30.020295 14175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:30.020311 14175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:30.025949 14175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:30.027107 14175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:30.028209 14175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:30.038300 14186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:30.038305 14185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:30.038314 14188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:30.038592 14175 server_base.cc:1034] running on GCE node
I20260810 07:29:30.039228 14175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:30.040426 14175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:30.041656 14175 hybrid_clock.cc:648] HybridClock initialized: now 1786346970041634 us; error 31 us; skew 500 ppm
Aug 10 07:29:30 dist-test-slave-ffrd krb5kdc[14158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:30.048580 14175 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 07:29:30.049707 14175 webserver.cc:460] Webserver started at http://127.0.0.1:37485/ using document root <none> and password file <none>
I20260810 07:29:30.050328 14175 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:30.050426 14175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:30.050676 14175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:30.052841 14175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "edbed2bdfd444300b0568794011924ca"
format_stamp: "Formatted at 2026-08-10 07:29:30 on dist-test-slave-ffrd"
server_key: "c19523bcec18262ec2c0f7fcdbdc886d"
server_key_iv: "7c7098b2bf64b88d203e9492c3c08864"
server_key_version: "encryptionkey@0"
I20260810 07:29:30.053484 14175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "edbed2bdfd444300b0568794011924ca"
format_stamp: "Formatted at 2026-08-10 07:29:30 on dist-test-slave-ffrd"
server_key: "c19523bcec18262ec2c0f7fcdbdc886d"
server_key_iv: "7c7098b2bf64b88d203e9492c3c08864"
server_key_version: "encryptionkey@0"
I20260810 07:29:30.057396 14175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 07:29:30.059759 14197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:30.061056 14175 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:29:30.061192 14175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "edbed2bdfd444300b0568794011924ca"
format_stamp: "Formatted at 2026-08-10 07:29:30 on dist-test-slave-ffrd"
server_key: "c19523bcec18262ec2c0f7fcdbdc886d"
server_key_iv: "7c7098b2bf64b88d203e9492c3c08864"
server_key_version: "encryptionkey@0"
I20260810 07:29:30.061303 14175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:30.066007 14175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:30.069284 14175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:30.069480 14175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:30.077037 14175 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36687
I20260810 07:29:30.077097 14288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36687 every 8 connection(s)
I20260810 07:29:30.077972 14175 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:30.080699 14289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:30.084918 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 14175
I20260810 07:29:30.085127 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:29:30.085422 12919 external_mini_cluster.cc:1428] Setting key ebbf0996c6320c04e8eaddd6f1f6a247
I20260810 07:29:30.086571 14289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edbed2bdfd444300b0568794011924ca: Bootstrap starting.
I20260810 07:29:30.088894 14289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edbed2bdfd444300b0568794011924ca: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:30.089862 14289 log.cc:824] T 00000000000000000000000000000000 P edbed2bdfd444300b0568794011924ca: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:30.090723 14293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:30.086822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58763 (local address 127.0.0.1:36687)
0810 07:29:30.087263 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:30.087277 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:30.087312 (+ 35us) server_negotiation.cc:407] Connection header received
0810 07:29:30.087917 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:30.087933 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:30.088196 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:30.088469 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:30.089472 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:30.089477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:30.089489 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:30.089500 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:30.089557 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:30.089576 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:30.089674 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:30.089851 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 07:29:30.090039 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":137,"threads_started":1}
I20260810 07:29:30.091856 14289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edbed2bdfd444300b0568794011924ca: No bootstrap required, opened a new log
I20260810 07:29:30.093685 12919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:29:30.096331 14289 raft_consensus.cc:348] T 00000000000000000000000000000000 P edbed2bdfd444300b0568794011924ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edbed2bdfd444300b0568794011924ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36687 } }
I20260810 07:29:30.096607 14289 raft_consensus.cc:374] T 00000000000000000000000000000000 P edbed2bdfd444300b0568794011924ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:30.096676 14289 raft_consensus.cc:729] T 00000000000000000000000000000000 P edbed2bdfd444300b0568794011924ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edbed2bdfd444300b0568794011924ca, State: Initialized, Role: FOLLOWER
I20260810 07:29:30.097205 14289 consensus_queue.cc:260] T 00000000000000000000000000000000 P edbed2bdfd444300b0568794011924ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edbed2bdfd444300b0568794011924ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36687 } }
I20260810 07:29:30.097383 14289 raft_consensus.cc:388] T 00000000000000000000000000000000 P edbed2bdfd444300b0568794011924ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:29:30.097430 14293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:30.095742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47497 (local address 127.0.0.1:36687)
0810 07:29:30.095851 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:30.095854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:30.096097 (+ 243us) server_negotiation.cc:407] Connection header received
0810 07:29:30.096173 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:30.096176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:30.096212 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:30.096288 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:30.097049 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:30.097052 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:30.097055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:30.097067 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:30.097097 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:30.097109 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:30.097191 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:30.097267 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 07:29:30.097307 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 07:29:30.097460 14289 raft_consensus.cc:482] T 00000000000000000000000000000000 P edbed2bdfd444300b0568794011924ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:29:30.097565 14289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edbed2bdfd444300b0568794011924ca [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:30.098698 14289 raft_consensus.cc:504] T 00000000000000000000000000000000 P edbed2bdfd444300b0568794011924ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edbed2bdfd444300b0568794011924ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36687 } }
I20260810 07:29:30.099056 14289 leader_election.cc:302] T 00000000000000000000000000000000 P edbed2bdfd444300b0568794011924ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edbed2bdfd444300b0568794011924ca; no voters:
I20260810 07:29:30.099412 14289 leader_election.cc:288] T 00000000000000000000000000000000 P edbed2bdfd444300b0568794011924ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:29:30.099509 14303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edbed2bdfd444300b0568794011924ca [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:30.099749 14303 raft_consensus.cc:686] T 00000000000000000000000000000000 P edbed2bdfd444300b0568794011924ca [term 1 LEADER]: Becoming Leader. State: Replica: edbed2bdfd444300b0568794011924ca, State: Running, Role: LEADER
I20260810 07:29:30.100232 14303 consensus_queue.cc:237] T 00000000000000000000000000000000 P edbed2bdfd444300b0568794011924ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edbed2bdfd444300b0568794011924ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36687 } }
I20260810 07:29:30.100725 14289 sys_catalog.cc:564] T 00000000000000000000000000000000 P edbed2bdfd444300b0568794011924ca [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:30.102126 14306 sys_catalog.cc:455] T 00000000000000000000000000000000 P edbed2bdfd444300b0568794011924ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edbed2bdfd444300b0568794011924ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edbed2bdfd444300b0568794011924ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36687 } } }
I20260810 07:29:30.102123 14307 sys_catalog.cc:455] T 00000000000000000000000000000000 P edbed2bdfd444300b0568794011924ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader edbed2bdfd444300b0568794011924ca. Latest consensus state: current_term: 1 leader_uuid: "edbed2bdfd444300b0568794011924ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edbed2bdfd444300b0568794011924ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36687 } } }
I20260810 07:29:30.102269 14307 sys_catalog.cc:458] T 00000000000000000000000000000000 P edbed2bdfd444300b0568794011924ca [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:30.102269 14306 sys_catalog.cc:458] T 00000000000000000000000000000000 P edbed2bdfd444300b0568794011924ca [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:30.103404 14321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:30.106137 14321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:30.111883 14321 catalog_manager.cc:1347] Generated new cluster ID: b84d83a6ba28489c9bb08a0eef5c8615
I20260810 07:29:30.111972 14321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:30.147049 14321 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:30.147486 14321 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:30.156492 14321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edbed2bdfd444300b0568794011924ca: Generated new TSK 0
I20260810 07:29:30.157471 14321 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:29:30.164595 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 14175
2026-08-10T07:29:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1669 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:29:30 dist-test-slave-ffrd krb5kdc[14343](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:29:30 dist-test-slave-ffrd krb5kdc[14343](info): set up 2 sockets
Aug 10 07:29:30 dist-test-slave-ffrd krb5kdc[14343](info): commencing operation
krb5kdc: starting...
W20260810 07:29:31.429390 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.238s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:29:31 dist-test-slave-ffrd krb5kdc[14343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:29:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 07:29:31.567600 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46131
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:31.683935 14366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:31.684253 14366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:31.684334 14366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:31.688382 14366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:31.688482 14366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:31.688522 14366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:31.688578 14366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:31.688617 14366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:31.693966 14366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:31.695068 14366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:31.696131 14366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:31.704916 14373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:31.705001 14381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:31.705104 14366 server_base.cc:1034] running on GCE node
W20260810 07:29:31.704916 14378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:31.705710 14366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:31.706799 14366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:31.708001 14366 hybrid_clock.cc:648] HybridClock initialized: now 1786346971707981 us; error 31 us; skew 500 ppm
Aug 10 07:29:31 dist-test-slave-ffrd krb5kdc[14343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:31.710942 14366 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 07:29:31.711824 14366 webserver.cc:460] Webserver started at http://127.0.0.1:41097/ using document root <none> and password file <none>
I20260810 07:29:31.712342 14366 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:31.712405 14366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:31.712574 14366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:31.714730 14366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "195585d29892406e8f95d5cbf6dafbb9"
format_stamp: "Formatted at 2026-08-10 07:29:31 on dist-test-slave-ffrd"
server_key: "1799a61df521203e2970220fef056d1f"
server_key_iv: "7e973215c96418424ed922f7a00f17a2"
server_key_version: "encryptionkey@0"
I20260810 07:29:31.715261 14366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "195585d29892406e8f95d5cbf6dafbb9"
format_stamp: "Formatted at 2026-08-10 07:29:31 on dist-test-slave-ffrd"
server_key: "1799a61df521203e2970220fef056d1f"
server_key_iv: "7e973215c96418424ed922f7a00f17a2"
server_key_version: "encryptionkey@0"
I20260810 07:29:31.719024 14366 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260810 07:29:31.721498 14390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:31.722700 14366 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:29:31.722846 14366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "195585d29892406e8f95d5cbf6dafbb9"
format_stamp: "Formatted at 2026-08-10 07:29:31 on dist-test-slave-ffrd"
server_key: "1799a61df521203e2970220fef056d1f"
server_key_iv: "7e973215c96418424ed922f7a00f17a2"
server_key_version: "encryptionkey@0"
I20260810 07:29:31.722966 14366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:31.742195 14366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:31.745525 14366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:31.745739 14366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:31.753347 14366 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46131
I20260810 07:29:31.753413 14475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46131 every 8 connection(s)
I20260810 07:29:31.754338 14366 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:31.755414 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 14366
I20260810 07:29:31.755569 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:29:31.755975 12919 external_mini_cluster.cc:1428] Setting key 3db38c37df0b0a14035a0825c52f4735
I20260810 07:29:31.757977 14480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:31.764057 14480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 195585d29892406e8f95d5cbf6dafbb9: Bootstrap starting.
Aug 10 07:29:31 dist-test-slave-ffrd krb5kdc[14343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 07:29:31.767055 14480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 195585d29892406e8f95d5cbf6dafbb9: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:31.767886 14480 log.cc:824] T 00000000000000000000000000000000 P 195585d29892406e8f95d5cbf6dafbb9: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:31.770560 14480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 195585d29892406e8f95d5cbf6dafbb9: No bootstrap required, opened a new log
I20260810 07:29:31.771266 14487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:31.757627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:59141 (local address 127.0.0.1:46131)
0810 07:29:31.758165 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:31.758181 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:31.758218 (+ 37us) server_negotiation.cc:407] Connection header received
0810 07:29:31.758847 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:31.758862 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:31.759190 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:31.759525 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:31.760334 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:31.761397 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:31.762397 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:31.762857 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:31.763038 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:31.766153 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:31.766158 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:31.766170 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:31.766197 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:31.768010 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:31.768776 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:31.768784 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:31.768792 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:31.769013 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:31.769392 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:31.769395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:31.769396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:31.769767 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:31.769935 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:31.770280 (+ 345us) server_negotiation.cc:299] Negotiation successful
0810 07:29:31.770507 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":163,"threads_started":1}
I20260810 07:29:31.774206 12919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:29:31.774755 14480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 195585d29892406e8f95d5cbf6dafbb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195585d29892406e8f95d5cbf6dafbb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46131 } }
I20260810 07:29:31.774982 14480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 195585d29892406e8f95d5cbf6dafbb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:31.775066 14480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 195585d29892406e8f95d5cbf6dafbb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 195585d29892406e8f95d5cbf6dafbb9, State: Initialized, Role: FOLLOWER
I20260810 07:29:31.775616 14480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 195585d29892406e8f95d5cbf6dafbb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195585d29892406e8f95d5cbf6dafbb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46131 } }
I20260810 07:29:31.775808 14480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 195585d29892406e8f95d5cbf6dafbb9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:29:31.775892 14480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 195585d29892406e8f95d5cbf6dafbb9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:29:31.776022 14480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 195585d29892406e8f95d5cbf6dafbb9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:31.777475 14480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 195585d29892406e8f95d5cbf6dafbb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195585d29892406e8f95d5cbf6dafbb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46131 } }
I20260810 07:29:31.777902 14480 leader_election.cc:302] T 00000000000000000000000000000000 P 195585d29892406e8f95d5cbf6dafbb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 195585d29892406e8f95d5cbf6dafbb9; no voters:
I20260810 07:29:31.778332 14480 leader_election.cc:288] T 00000000000000000000000000000000 P 195585d29892406e8f95d5cbf6dafbb9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:29:31.778409 14496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 195585d29892406e8f95d5cbf6dafbb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:31.778641 14496 raft_consensus.cc:686] T 00000000000000000000000000000000 P 195585d29892406e8f95d5cbf6dafbb9 [term 1 LEADER]: Becoming Leader. State: Replica: 195585d29892406e8f95d5cbf6dafbb9, State: Running, Role: LEADER
I20260810 07:29:31.779034 14496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 195585d29892406e8f95d5cbf6dafbb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195585d29892406e8f95d5cbf6dafbb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46131 } }
I20260810 07:29:31.779762 14480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 195585d29892406e8f95d5cbf6dafbb9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:31.780339 14496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 195585d29892406e8f95d5cbf6dafbb9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 195585d29892406e8f95d5cbf6dafbb9. Latest consensus state: current_term: 1 leader_uuid: "195585d29892406e8f95d5cbf6dafbb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195585d29892406e8f95d5cbf6dafbb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46131 } } }
I20260810 07:29:31.781605 14496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 195585d29892406e8f95d5cbf6dafbb9 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:31.781468 14497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 195585d29892406e8f95d5cbf6dafbb9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "195585d29892406e8f95d5cbf6dafbb9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "195585d29892406e8f95d5cbf6dafbb9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46131 } } }
I20260810 07:29:31.782439 14497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 195585d29892406e8f95d5cbf6dafbb9 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:31.782454 14516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:31.786499 14516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:31.786813 14487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:31.776497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:51979 (local address 127.0.0.1:46131)
0810 07:29:31.776615 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:31.776619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:31.776798 (+ 179us) server_negotiation.cc:407] Connection header received
0810 07:29:31.776919 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:31.776921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:31.777012 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:31.777088 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:31.778006 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:31.778898 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:31.779905 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:31.780199 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:31.780738 (+ 539us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:31.782225 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:31.782229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:31.782232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:31.782258 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:31.784129 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:31.784804 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:31.784808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:31.784810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:31.784907 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:31.785386 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:31.785389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:31.785391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:31.785632 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:31.786422 (+ 790us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:31.786536 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 07:29:31.786648 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 07:29:31.792764 14516 catalog_manager.cc:1347] Generated new cluster ID: 823a190729ac48108cb91033483ce7fc
I20260810 07:29:31.792892 14516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:31.805456 14516 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:31.806478 14516 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:31.819422 14516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 195585d29892406e8f95d5cbf6dafbb9: Generated new TSK 0
I20260810 07:29:31.820324 14516 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:29:31.824828 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 14366
W20260810 07:29:31.842427 14341 connection.cc:535] client connection to 127.0.0.1:46131 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46131): Connection reset by peer (error 104)
2026-08-10T07:29:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1665 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:29:31 dist-test-slave-ffrd krb5kdc[14528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:29:31 dist-test-slave-ffrd krb5kdc[14528](info): set up 2 sockets
Aug 10 07:29:31 dist-test-slave-ffrd krb5kdc[14528](info): commencing operation
krb5kdc: starting...
W20260810 07:29:33.118026 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.261s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:29:33 dist-test-slave-ffrd krb5kdc[14528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:29:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 07:29:33.251992 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37939
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:33.362493 14551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:33.362782 14551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:33.362855 14551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:33.366772 14551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:33.366861 14551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:33.366915 14551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:33.366950 14551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:33.366984 14551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:33.372599 14551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:33.373667 14551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:33.374650 14551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:33.383599 14561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:33.383661 14551 server_base.cc:1034] running on GCE node
W20260810 07:29:33.383673 14559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:33.383615 14558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:33.384375 14551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:33.385509 14551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:33.386706 14551 hybrid_clock.cc:648] HybridClock initialized: now 1786346973386679 us; error 38 us; skew 500 ppm
Aug 10 07:29:33 dist-test-slave-ffrd krb5kdc[14528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:33.389905 14551 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 07:29:33.390846 14551 webserver.cc:460] Webserver started at http://127.0.0.1:41877/ using document root <none> and password file <none>
I20260810 07:29:33.391455 14551 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:33.391557 14551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:33.391814 14551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:33.394057 14551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "a94476f034bd411cbb67931d8ef03fdb"
format_stamp: "Formatted at 2026-08-10 07:29:33 on dist-test-slave-ffrd"
server_key: "66b0b7e16990325d75e204a9c0d4f71e"
server_key_iv: "b2108154e823fd49afea772e996f581e"
server_key_version: "encryptionkey@0"
I20260810 07:29:33.394680 14551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "a94476f034bd411cbb67931d8ef03fdb"
format_stamp: "Formatted at 2026-08-10 07:29:33 on dist-test-slave-ffrd"
server_key: "66b0b7e16990325d75e204a9c0d4f71e"
server_key_iv: "b2108154e823fd49afea772e996f581e"
server_key_version: "encryptionkey@0"
I20260810 07:29:33.398674 14551 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 07:29:33.401311 14572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:33.402637 14551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 07:29:33.402789 14551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "a94476f034bd411cbb67931d8ef03fdb"
format_stamp: "Formatted at 2026-08-10 07:29:33 on dist-test-slave-ffrd"
server_key: "66b0b7e16990325d75e204a9c0d4f71e"
server_key_iv: "b2108154e823fd49afea772e996f581e"
server_key_version: "encryptionkey@0"
I20260810 07:29:33.402923 14551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:33.413029 14551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:33.416417 14551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:33.416635 14551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:33.423905 14551 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37939
I20260810 07:29:33.423969 14654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37939 every 8 connection(s)
I20260810 07:29:33.424916 14551 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:33.427774 14655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:33.428665 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 14551
I20260810 07:29:33.428825 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:29:33.429147 12919 external_mini_cluster.cc:1428] Setting key 4c9a9dcb43ba18775fc82e83eafedd34
I20260810 07:29:33.434558 14655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a94476f034bd411cbb67931d8ef03fdb: Bootstrap starting.
I20260810 07:29:33.437206 14655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a94476f034bd411cbb67931d8ef03fdb: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:33.438025 14655 log.cc:824] T 00000000000000000000000000000000 P a94476f034bd411cbb67931d8ef03fdb: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:33.438277 14660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:33.430505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:52619 (local address 127.0.0.1:37939)
0810 07:29:33.430945 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:33.430961 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:33.430997 (+ 36us) server_negotiation.cc:407] Connection header received
0810 07:29:33.431645 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:33.431661 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:33.432007 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:33.432295 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:33.432653 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:33.433714 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:33.434896 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:33.435374 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:33.435496 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:33.436442 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:33.436447 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:33.436459 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:33.436843 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:33.436903 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:33.436926 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:33.437097 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:33.437379 (+ 282us) server_negotiation.cc:299] Negotiation successful
0810 07:29:33.437499 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":150,"threads_started":1}
I20260810 07:29:33.439985 14655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a94476f034bd411cbb67931d8ef03fdb: No bootstrap required, opened a new log
I20260810 07:29:33.441318 12919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:29:33.444774 14655 raft_consensus.cc:348] T 00000000000000000000000000000000 P a94476f034bd411cbb67931d8ef03fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94476f034bd411cbb67931d8ef03fdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37939 } }
I20260810 07:29:33.445080 14655 raft_consensus.cc:374] T 00000000000000000000000000000000 P a94476f034bd411cbb67931d8ef03fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:33.445160 14655 raft_consensus.cc:729] T 00000000000000000000000000000000 P a94476f034bd411cbb67931d8ef03fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a94476f034bd411cbb67931d8ef03fdb, State: Initialized, Role: FOLLOWER
I20260810 07:29:33.445636 14655 consensus_queue.cc:260] T 00000000000000000000000000000000 P a94476f034bd411cbb67931d8ef03fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94476f034bd411cbb67931d8ef03fdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37939 } }
I20260810 07:29:33.445811 14655 raft_consensus.cc:388] T 00000000000000000000000000000000 P a94476f034bd411cbb67931d8ef03fdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:29:33.445883 14655 raft_consensus.cc:482] T 00000000000000000000000000000000 P a94476f034bd411cbb67931d8ef03fdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:29:33.445997 14655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a94476f034bd411cbb67931d8ef03fdb [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:33.447234 14655 raft_consensus.cc:504] T 00000000000000000000000000000000 P a94476f034bd411cbb67931d8ef03fdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94476f034bd411cbb67931d8ef03fdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37939 } }
I20260810 07:29:33.447641 14655 leader_election.cc:302] T 00000000000000000000000000000000 P a94476f034bd411cbb67931d8ef03fdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a94476f034bd411cbb67931d8ef03fdb; no voters:
I20260810 07:29:33.448040 14655 leader_election.cc:288] T 00000000000000000000000000000000 P a94476f034bd411cbb67931d8ef03fdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:29:33.448130 14675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a94476f034bd411cbb67931d8ef03fdb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:33.448343 14675 raft_consensus.cc:686] T 00000000000000000000000000000000 P a94476f034bd411cbb67931d8ef03fdb [term 1 LEADER]: Becoming Leader. State: Replica: a94476f034bd411cbb67931d8ef03fdb, State: Running, Role: LEADER
I20260810 07:29:33.448621 14660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:33.443864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:55687 (local address 127.0.0.1:37939)
0810 07:29:33.443999 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:33.444003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:33.444140 (+ 137us) server_negotiation.cc:407] Connection header received
0810 07:29:33.444259 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:33.444261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:33.444302 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:33.444386 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:33.444703 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:33.445486 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:33.446626 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:33.446892 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:33.446976 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:33.447972 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:33.447975 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:33.447978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:33.447990 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:33.448014 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:33.448025 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:33.448134 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:33.448466 (+ 332us) server_negotiation.cc:299] Negotiation successful
0810 07:29:33.448506 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 07:29:33.448704 14675 consensus_queue.cc:237] T 00000000000000000000000000000000 P a94476f034bd411cbb67931d8ef03fdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94476f034bd411cbb67931d8ef03fdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37939 } }
I20260810 07:29:33.449505 14655 sys_catalog.cc:564] T 00000000000000000000000000000000 P a94476f034bd411cbb67931d8ef03fdb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:33.450028 14675 sys_catalog.cc:455] T 00000000000000000000000000000000 P a94476f034bd411cbb67931d8ef03fdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a94476f034bd411cbb67931d8ef03fdb. Latest consensus state: current_term: 1 leader_uuid: "a94476f034bd411cbb67931d8ef03fdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94476f034bd411cbb67931d8ef03fdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37939 } } }
I20260810 07:29:33.450148 14675 sys_catalog.cc:458] T 00000000000000000000000000000000 P a94476f034bd411cbb67931d8ef03fdb [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:33.450387 14675 sys_catalog.cc:455] T 00000000000000000000000000000000 P a94476f034bd411cbb67931d8ef03fdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a94476f034bd411cbb67931d8ef03fdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a94476f034bd411cbb67931d8ef03fdb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37939 } } }
I20260810 07:29:33.450551 14675 sys_catalog.cc:458] T 00000000000000000000000000000000 P a94476f034bd411cbb67931d8ef03fdb [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:33.450579 14686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:33.453959 14686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:33.459723 14686 catalog_manager.cc:1347] Generated new cluster ID: 85672add7b5b4a56aac3922aa4f45d87
I20260810 07:29:33.459811 14686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:33.471994 14686 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:33.473105 14686 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:33.479218 14686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a94476f034bd411cbb67931d8ef03fdb: Generated new TSK 0
I20260810 07:29:33.479959 14686 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:29:33.488479 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 14551
W20260810 07:29:33.500741 14525 connection.cc:535] client connection to 127.0.0.1:37939 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37939): Connection reset by peer (error 104)
2026-08-10T07:29:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1658 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:29:33 dist-test-slave-ffrd krb5kdc[14709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:29:33 dist-test-slave-ffrd krb5kdc[14709](info): set up 2 sockets
Aug 10 07:29:33 dist-test-slave-ffrd krb5kdc[14709](info): commencing operation
krb5kdc: starting...
W20260810 07:29:34.757474 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.242s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:29:34 dist-test-slave-ffrd krb5kdc[14709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:29:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 07:29:34.895952 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36475
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:35.012756 14730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:35.013134 14730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:35.013221 14730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:35.017294 14730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:35.017402 14730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:35.017442 14730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:35.017503 14730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:35.017544 14730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:35.022929 14730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:35.024127 14730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:35.025313 14730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:35.033962 14744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:35.033957 14741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:35.033952 14740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:35.034365 14730 server_base.cc:1034] running on GCE node
I20260810 07:29:35.034905 14730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:35.035934 14730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:35.037093 14730 hybrid_clock.cc:648] HybridClock initialized: now 1786346975037063 us; error 41 us; skew 500 ppm
Aug 10 07:29:35 dist-test-slave-ffrd krb5kdc[14709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:35.040014 14730 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 07:29:35.041008 14730 webserver.cc:460] Webserver started at http://127.0.0.1:35523/ using document root <none> and password file <none>
I20260810 07:29:35.041559 14730 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:35.041620 14730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:35.041783 14730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:35.043816 14730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "515d3b5e7e6040198d17b1667eeff4dc"
format_stamp: "Formatted at 2026-08-10 07:29:35 on dist-test-slave-ffrd"
server_key: "718534a5c611d92bd41840b04a2ca2c1"
server_key_iv: "50218598aeacaf5b371cedd269b7499c"
server_key_version: "encryptionkey@0"
I20260810 07:29:35.044330 14730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "515d3b5e7e6040198d17b1667eeff4dc"
format_stamp: "Formatted at 2026-08-10 07:29:35 on dist-test-slave-ffrd"
server_key: "718534a5c611d92bd41840b04a2ca2c1"
server_key_iv: "50218598aeacaf5b371cedd269b7499c"
server_key_version: "encryptionkey@0"
I20260810 07:29:35.048210 14730 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 07:29:35.050843 14753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:35.052124 14730 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 07:29:35.052228 14730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "515d3b5e7e6040198d17b1667eeff4dc"
format_stamp: "Formatted at 2026-08-10 07:29:35 on dist-test-slave-ffrd"
server_key: "718534a5c611d92bd41840b04a2ca2c1"
server_key_iv: "50218598aeacaf5b371cedd269b7499c"
server_key_version: "encryptionkey@0"
I20260810 07:29:35.052316 14730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:35.056902 14730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:35.065220 14730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:35.065383 14730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:35.072415 14730 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36475
I20260810 07:29:35.072436 14833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36475 every 8 connection(s)
I20260810 07:29:35.073447 14730 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:35.073995 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 14730
I20260810 07:29:35.074141 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:29:35.074461 12919 external_mini_cluster.cc:1428] Setting key 5baf1e8fec3bf301fe326a9a600688eb
I20260810 07:29:35.076740 14834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:35.079702 14838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:35.075887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:57823 (local address 127.0.0.1:36475)
0810 07:29:35.076331 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:35.076346 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:35.076383 (+ 37us) server_negotiation.cc:407] Connection header received
0810 07:29:35.077207 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:35.077225 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:35.077492 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:35.077825 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:35.078578 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:35.078590 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:35.078602 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:35.078612 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:35.078660 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:35.078677 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:35.078826 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:35.078903 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 07:29:35.079065 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":138,"threads_started":1}
I20260810 07:29:35.082540 12919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:29:35.083094 14834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 515d3b5e7e6040198d17b1667eeff4dc: Bootstrap starting.
I20260810 07:29:35.085615 14834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 515d3b5e7e6040198d17b1667eeff4dc: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:35.086387 14838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:35.084629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:47323 (local address 127.0.0.1:36475)
0810 07:29:35.084760 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:35.084764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:35.085003 (+ 239us) server_negotiation.cc:407] Connection header received
0810 07:29:35.085103 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:35.085106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:35.085135 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:35.085207 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:35.085975 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:35.085978 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:35.085980 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:35.085991 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:35.086016 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:35.086026 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:35.086092 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:35.086208 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 07:29:35.086239 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 07:29:35.086607 14834 log.cc:824] T 00000000000000000000000000000000 P 515d3b5e7e6040198d17b1667eeff4dc: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:35.088536 14834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 515d3b5e7e6040198d17b1667eeff4dc: No bootstrap required, opened a new log
I20260810 07:29:35.092015 14834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 515d3b5e7e6040198d17b1667eeff4dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515d3b5e7e6040198d17b1667eeff4dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36475 } }
I20260810 07:29:35.092255 14834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 515d3b5e7e6040198d17b1667eeff4dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:35.092340 14834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 515d3b5e7e6040198d17b1667eeff4dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 515d3b5e7e6040198d17b1667eeff4dc, State: Initialized, Role: FOLLOWER
I20260810 07:29:35.092796 14834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 515d3b5e7e6040198d17b1667eeff4dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515d3b5e7e6040198d17b1667eeff4dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36475 } }
I20260810 07:29:35.093003 14834 raft_consensus.cc:388] T 00000000000000000000000000000000 P 515d3b5e7e6040198d17b1667eeff4dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:29:35.093087 14834 raft_consensus.cc:482] T 00000000000000000000000000000000 P 515d3b5e7e6040198d17b1667eeff4dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:29:35.093232 14834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 515d3b5e7e6040198d17b1667eeff4dc [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:35.094213 14834 raft_consensus.cc:504] T 00000000000000000000000000000000 P 515d3b5e7e6040198d17b1667eeff4dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515d3b5e7e6040198d17b1667eeff4dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36475 } }
I20260810 07:29:35.094543 14834 leader_election.cc:302] T 00000000000000000000000000000000 P 515d3b5e7e6040198d17b1667eeff4dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 515d3b5e7e6040198d17b1667eeff4dc; no voters:
I20260810 07:29:35.094871 14834 leader_election.cc:288] T 00000000000000000000000000000000 P 515d3b5e7e6040198d17b1667eeff4dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:29:35.095041 14851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 515d3b5e7e6040198d17b1667eeff4dc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:35.095297 14851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 515d3b5e7e6040198d17b1667eeff4dc [term 1 LEADER]: Becoming Leader. State: Replica: 515d3b5e7e6040198d17b1667eeff4dc, State: Running, Role: LEADER
I20260810 07:29:35.095639 14851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 515d3b5e7e6040198d17b1667eeff4dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515d3b5e7e6040198d17b1667eeff4dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36475 } }
I20260810 07:29:35.096112 14834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 515d3b5e7e6040198d17b1667eeff4dc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:35.097373 14852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 515d3b5e7e6040198d17b1667eeff4dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "515d3b5e7e6040198d17b1667eeff4dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515d3b5e7e6040198d17b1667eeff4dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36475 } } }
I20260810 07:29:35.097499 14852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 515d3b5e7e6040198d17b1667eeff4dc [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:35.097909 14868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:35.097915 14853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 515d3b5e7e6040198d17b1667eeff4dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 515d3b5e7e6040198d17b1667eeff4dc. Latest consensus state: current_term: 1 leader_uuid: "515d3b5e7e6040198d17b1667eeff4dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515d3b5e7e6040198d17b1667eeff4dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36475 } } }
I20260810 07:29:35.098120 14853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 515d3b5e7e6040198d17b1667eeff4dc [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:35.100898 14868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:35.106700 14868 catalog_manager.cc:1347] Generated new cluster ID: bd9c3d903ae54441bc148f1d91e93972
I20260810 07:29:35.106784 14868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:35.133706 14868 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:35.134079 14868 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:35.139812 14868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 515d3b5e7e6040198d17b1667eeff4dc: Generated new TSK 0
I20260810 07:29:35.140635 14868 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:29:35.144255 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 14730
2026-08-10T07:29:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1659 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:29:35 dist-test-slave-ffrd krb5kdc[14887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:29:35 dist-test-slave-ffrd krb5kdc[14887](info): set up 2 sockets
Aug 10 07:29:35 dist-test-slave-ffrd krb5kdc[14887](info): commencing operation
krb5kdc: starting...
W20260810 07:29:36.439442 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.264s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:29:36 dist-test-slave-ffrd krb5kdc[14887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:29:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 07:29:36.580214 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34821
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:36.695345 14908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:36.695768 14908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:36.695854 14908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:36.699966 14908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:36.700073 14908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:36.700112 14908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:36.700172 14908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:36.700213 14908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:36.705659 14908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34821
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:36.706813 14908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:36.707872 14908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:36.717113 14922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:36.717147 14914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:36.717293 14908 server_base.cc:1034] running on GCE node
W20260810 07:29:36.717124 14918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:36.718024 14908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:36.719206 14908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:36.720398 14908 hybrid_clock.cc:648] HybridClock initialized: now 1786346976720378 us; error 34 us; skew 500 ppm
Aug 10 07:29:36 dist-test-slave-ffrd krb5kdc[14887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:36.723337 14908 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 07:29:36.724267 14908 webserver.cc:460] Webserver started at http://127.0.0.1:43169/ using document root <none> and password file <none>
I20260810 07:29:36.725044 14908 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:36.725121 14908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:36.725359 14908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:36.727490 14908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "252721744f46405c9aa2cd022282787d"
format_stamp: "Formatted at 2026-08-10 07:29:36 on dist-test-slave-ffrd"
server_key: "fe47a023413b2167d6bffddd6a80636f"
server_key_iv: "d0a8a2ea1abaae478328a10422d6d4a5"
server_key_version: "encryptionkey@0"
I20260810 07:29:36.728096 14908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "252721744f46405c9aa2cd022282787d"
format_stamp: "Formatted at 2026-08-10 07:29:36 on dist-test-slave-ffrd"
server_key: "fe47a023413b2167d6bffddd6a80636f"
server_key_iv: "d0a8a2ea1abaae478328a10422d6d4a5"
server_key_version: "encryptionkey@0"
I20260810 07:29:36.731992 14908 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 07:29:36.734519 14935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:36.735805 14908 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 07:29:36.735960 14908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "252721744f46405c9aa2cd022282787d"
format_stamp: "Formatted at 2026-08-10 07:29:36 on dist-test-slave-ffrd"
server_key: "fe47a023413b2167d6bffddd6a80636f"
server_key_iv: "d0a8a2ea1abaae478328a10422d6d4a5"
server_key_version: "encryptionkey@0"
I20260810 07:29:36.736074 14908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:36.763055 14908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:36.766181 14908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:36.766408 14908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:36.774253 14908 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34821
I20260810 07:29:36.774276 15027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34821 every 8 connection(s)
I20260810 07:29:36.775292 14908 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:36.777756 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 14908
I20260810 07:29:36.777925 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:29:36.778208 12919 external_mini_cluster.cc:1428] Setting key d46d8a096b110b4dfc95d7f740aa4945
I20260810 07:29:36.778553 15029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:36.786149 15029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 252721744f46405c9aa2cd022282787d: Bootstrap starting.
Aug 10 07:29:36 dist-test-slave-ffrd krb5kdc[14887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 07:29:36.789664 15029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 252721744f46405c9aa2cd022282787d: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:36.790685 15029 log.cc:824] T 00000000000000000000000000000000 P 252721744f46405c9aa2cd022282787d: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:36.793349 15029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 252721744f46405c9aa2cd022282787d: No bootstrap required, opened a new log
I20260810 07:29:36.795809 15034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:36.779580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:55297 (local address 127.0.0.1:34821)
0810 07:29:36.780008 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:36.780024 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:36.780059 (+ 35us) server_negotiation.cc:407] Connection header received
0810 07:29:36.780716 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:36.780732 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:36.781141 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:36.781420 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:36.782601 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:36.783640 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:36.785043 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:36.785728 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:36.785909 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:36.789539 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:36.789548 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:36.789567 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:36.789633 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:36.791856 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:36.792504 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:36.792509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:36.792514 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:36.792625 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:36.793343 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:36.793348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:36.793351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:36.793975 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:36.794235 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:36.794588 (+ 353us) server_negotiation.cc:299] Negotiation successful
0810 07:29:36.794854 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":135,"threads_started":1}
I20260810 07:29:36.798796 15029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 252721744f46405c9aa2cd022282787d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252721744f46405c9aa2cd022282787d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34821 } }
I20260810 07:29:36.799095 15029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 252721744f46405c9aa2cd022282787d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:36.799149 15029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 252721744f46405c9aa2cd022282787d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 252721744f46405c9aa2cd022282787d, State: Initialized, Role: FOLLOWER
I20260810 07:29:36.799672 15029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 252721744f46405c9aa2cd022282787d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252721744f46405c9aa2cd022282787d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34821 } }
I20260810 07:29:36.799850 15029 raft_consensus.cc:388] T 00000000000000000000000000000000 P 252721744f46405c9aa2cd022282787d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:29:36.799926 15029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 252721744f46405c9aa2cd022282787d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:29:36.800081 15029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 252721744f46405c9aa2cd022282787d [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:36.801105 15029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 252721744f46405c9aa2cd022282787d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252721744f46405c9aa2cd022282787d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34821 } }
I20260810 07:29:36.801501 15029 leader_election.cc:302] T 00000000000000000000000000000000 P 252721744f46405c9aa2cd022282787d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 252721744f46405c9aa2cd022282787d; no voters:
I20260810 07:29:36.801699 12919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:29:36.801934 15029 leader_election.cc:288] T 00000000000000000000000000000000 P 252721744f46405c9aa2cd022282787d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:29:36.802601 15040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 252721744f46405c9aa2cd022282787d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:36.802841 15040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 252721744f46405c9aa2cd022282787d [term 1 LEADER]: Becoming Leader. State: Replica: 252721744f46405c9aa2cd022282787d, State: Running, Role: LEADER
I20260810 07:29:36.803186 15040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 252721744f46405c9aa2cd022282787d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252721744f46405c9aa2cd022282787d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34821 } }
I20260810 07:29:36.805330 15040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 252721744f46405c9aa2cd022282787d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 252721744f46405c9aa2cd022282787d. Latest consensus state: current_term: 1 leader_uuid: "252721744f46405c9aa2cd022282787d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252721744f46405c9aa2cd022282787d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34821 } } }
I20260810 07:29:36.805549 15040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 252721744f46405c9aa2cd022282787d [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:36.805562 15039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 252721744f46405c9aa2cd022282787d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "252721744f46405c9aa2cd022282787d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "252721744f46405c9aa2cd022282787d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34821 } } }
I20260810 07:29:36.805804 15039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 252721744f46405c9aa2cd022282787d [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:36.808223 15055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:36.803573 15029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 252721744f46405c9aa2cd022282787d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:36.811631 15055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:36.817584 15055 catalog_manager.cc:1347] Generated new cluster ID: 261931549f8140d1b68e4dd35db28652
I20260810 07:29:36.817691 15055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:36.822415 15034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:36.805492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:35195 (local address 127.0.0.1:34821)
0810 07:29:36.805660 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:36.805665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:36.806136 (+ 471us) server_negotiation.cc:407] Connection header received
0810 07:29:36.806263 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:36.806269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:36.806330 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:36.806409 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:36.807258 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:36.808442 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:36.809639 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:36.810002 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:36.810078 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:36.814551 (+ 4473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:36.814554 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:36.814557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:36.814584 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:36.819347 (+ 4763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:36.819954 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:36.819957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:36.819959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:36.820039 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:36.820371 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:36.820375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:36.820376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:36.820570 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:36.820689 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:36.822156 (+ 1467us) server_negotiation.cc:299] Negotiation successful
0810 07:29:36.822255 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":78}
I20260810 07:29:36.837852 15055 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:36.838851 15055 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:36.845616 15055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 252721744f46405c9aa2cd022282787d: Generated new TSK 0
I20260810 07:29:36.846347 15055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:29:36.851198 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 14908
W20260810 07:29:36.866638 14885 connection.cc:535] client connection to 127.0.0.1:34821 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34821): Connection reset by peer (error 104)
2026-08-10T07:29:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1705 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:29:36 dist-test-slave-ffrd krb5kdc[15084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:29:36 dist-test-slave-ffrd krb5kdc[15084](info): set up 2 sockets
Aug 10 07:29:36 dist-test-slave-ffrd krb5kdc[15084](info): commencing operation
krb5kdc: starting...
W20260810 07:29:38.131783 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.250s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:29:38 dist-test-slave-ffrd krb5kdc[15084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:29:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 07:29:38.270237 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40707
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:38.405898 15107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:38.406252 15107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:38.406334 15107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:38.410550 15107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:38.410657 15107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:38.410698 15107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:38.410796 15107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:38.410851 15107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:38.416584 15107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40707
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:38.417855 15107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:38.418953 15107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:38.427946 15118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:38.427978 15115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:38.427954 15114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:38.428121 15107 server_base.cc:1034] running on GCE node
I20260810 07:29:38.428870 15107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:38.430063 15107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:38.431313 15107 hybrid_clock.cc:648] HybridClock initialized: now 1786346978431288 us; error 33 us; skew 500 ppm
Aug 10 07:29:38 dist-test-slave-ffrd krb5kdc[15084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:38.434651 15107 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 07:29:38.435565 15107 webserver.cc:460] Webserver started at http://127.0.0.1:46209/ using document root <none> and password file <none>
I20260810 07:29:38.436110 15107 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:38.436172 15107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:38.436352 15107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:38.438583 15107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "5e7ef2e37a3e49d09ab341374f53ba0e"
format_stamp: "Formatted at 2026-08-10 07:29:38 on dist-test-slave-ffrd"
server_key: "fb4da4e8932f81e98d75c486aa52eb9c"
server_key_iv: "cdde5bc124a6031b6f03816c201b5ced"
server_key_version: "encryptionkey@0"
I20260810 07:29:38.439119 15107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "5e7ef2e37a3e49d09ab341374f53ba0e"
format_stamp: "Formatted at 2026-08-10 07:29:38 on dist-test-slave-ffrd"
server_key: "fb4da4e8932f81e98d75c486aa52eb9c"
server_key_iv: "cdde5bc124a6031b6f03816c201b5ced"
server_key_version: "encryptionkey@0"
I20260810 07:29:38.443096 15107 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 07:29:38.445704 15132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:38.447046 15107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 07:29:38.447160 15107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "5e7ef2e37a3e49d09ab341374f53ba0e"
format_stamp: "Formatted at 2026-08-10 07:29:38 on dist-test-slave-ffrd"
server_key: "fb4da4e8932f81e98d75c486aa52eb9c"
server_key_iv: "cdde5bc124a6031b6f03816c201b5ced"
server_key_version: "encryptionkey@0"
I20260810 07:29:38.447252 15107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:38.468770 15107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:38.472379 15107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:38.472561 15107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:38.480923 15107 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40707
I20260810 07:29:38.480993 15227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40707 every 8 connection(s)
I20260810 07:29:38.482024 15107 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:38.485121 15234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:38.489178 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 15107
I20260810 07:29:38.489334 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:29:38.489666 12919 external_mini_cluster.cc:1428] Setting key d1678ec2b905abc3a75feeac8078c1b6
I20260810 07:29:38.491657 15234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7ef2e37a3e49d09ab341374f53ba0e: Bootstrap starting.
I20260810 07:29:38.494141 15234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e7ef2e37a3e49d09ab341374f53ba0e: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:38.494962 15234 log.cc:824] T 00000000000000000000000000000000 P 5e7ef2e37a3e49d09ab341374f53ba0e: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:38.497071 15234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7ef2e37a3e49d09ab341374f53ba0e: No bootstrap required, opened a new log
I20260810 07:29:38.500527 15237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:38.491332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:52167 (local address 127.0.0.1:40707)
0810 07:29:38.491871 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:38.491905 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:38.491945 (+ 40us) server_negotiation.cc:407] Connection header received
0810 07:29:38.492750 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:38.492768 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:38.493227 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:38.493597 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:38.494042 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:38.495211 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:38.496158 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:38.496651 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:38.496833 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:38.498144 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:38.498149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:38.498297 (+ 148us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:38.498963 (+ 666us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:38.499149 (+ 186us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:38.499200 (+ 51us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:38.499395 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:38.499596 (+ 201us) server_negotiation.cc:299] Negotiation successful
0810 07:29:38.499754 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":145,"threads_started":1}
I20260810 07:29:38.501443 15234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e7ef2e37a3e49d09ab341374f53ba0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7ef2e37a3e49d09ab341374f53ba0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40707 } }
I20260810 07:29:38.501732 15234 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e7ef2e37a3e49d09ab341374f53ba0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:38.501838 15234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e7ef2e37a3e49d09ab341374f53ba0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7ef2e37a3e49d09ab341374f53ba0e, State: Initialized, Role: FOLLOWER
I20260810 07:29:38.502275 15234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e7ef2e37a3e49d09ab341374f53ba0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7ef2e37a3e49d09ab341374f53ba0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40707 } }
I20260810 07:29:38.502398 15234 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e7ef2e37a3e49d09ab341374f53ba0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:29:38.502483 15234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e7ef2e37a3e49d09ab341374f53ba0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:29:38.502609 15234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e7ef2e37a3e49d09ab341374f53ba0e [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:38.503741 15234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e7ef2e37a3e49d09ab341374f53ba0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7ef2e37a3e49d09ab341374f53ba0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40707 } }
I20260810 07:29:38.504098 15234 leader_election.cc:302] T 00000000000000000000000000000000 P 5e7ef2e37a3e49d09ab341374f53ba0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e7ef2e37a3e49d09ab341374f53ba0e; no voters:
I20260810 07:29:38.504436 12919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:29:38.504534 15234 leader_election.cc:288] T 00000000000000000000000000000000 P 5e7ef2e37a3e49d09ab341374f53ba0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:29:38.504633 15240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e7ef2e37a3e49d09ab341374f53ba0e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:38.504860 15240 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e7ef2e37a3e49d09ab341374f53ba0e [term 1 LEADER]: Becoming Leader. State: Replica: 5e7ef2e37a3e49d09ab341374f53ba0e, State: Running, Role: LEADER
I20260810 07:29:38.505303 15240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e7ef2e37a3e49d09ab341374f53ba0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7ef2e37a3e49d09ab341374f53ba0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40707 } }
I20260810 07:29:38.505837 15234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e7ef2e37a3e49d09ab341374f53ba0e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:38.506793 15240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7ef2e37a3e49d09ab341374f53ba0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e7ef2e37a3e49d09ab341374f53ba0e. Latest consensus state: current_term: 1 leader_uuid: "5e7ef2e37a3e49d09ab341374f53ba0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7ef2e37a3e49d09ab341374f53ba0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40707 } } }
I20260810 07:29:38.506938 15240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7ef2e37a3e49d09ab341374f53ba0e [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:38.507652 15256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:38.507616 15245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7ef2e37a3e49d09ab341374f53ba0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e7ef2e37a3e49d09ab341374f53ba0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e7ef2e37a3e49d09ab341374f53ba0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40707 } } }
I20260810 07:29:38.507886 15245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7ef2e37a3e49d09ab341374f53ba0e [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:38.511572 15256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:38.513382 15237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:38.507359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:35243 (local address 127.0.0.1:40707)
0810 07:29:38.507451 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:38.507457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:38.507471 (+ 14us) server_negotiation.cc:407] Connection header received
0810 07:29:38.507511 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:38.507514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:38.507564 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:38.507641 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:38.508057 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:38.509654 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:38.510888 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:38.511229 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:38.511353 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:38.512539 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:38.512543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:38.512546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:38.512561 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:38.512587 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:38.512600 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:38.512684 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:38.513192 (+ 508us) server_negotiation.cc:299] Negotiation successful
0810 07:29:38.513230 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 07:29:38.517961 15256 catalog_manager.cc:1347] Generated new cluster ID: 49a1231fca9c412b891819890cac6d04
I20260810 07:29:38.518044 15256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:38.534415 15256 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:38.535494 15256 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:38.542697 15256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e7ef2e37a3e49d09ab341374f53ba0e: Generated new TSK 0
I20260810 07:29:38.543846 15256 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:29:38.556068 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 15107
W20260810 07:29:38.575207 15082 connection.cc:535] client connection to 127.0.0.1:40707 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40707): Connection reset by peer (error 104)
2026-08-10T07:29:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1708 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:29:38 dist-test-slave-ffrd krb5kdc[15283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:29:38 dist-test-slave-ffrd krb5kdc[15283](info): set up 2 sockets
Aug 10 07:29:38 dist-test-slave-ffrd krb5kdc[15283](info): commencing operation
krb5kdc: starting...
W20260810 07:29:39.845920 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.256s user 0.004s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:29:39 dist-test-slave-ffrd krb5kdc[15283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:29:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 07:29:39.984267 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33945
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:40.100613 15304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:40.100991 15304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:40.101080 15304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:40.105158 15304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:40.105252 15304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:40.105288 15304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:40.105326 15304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:40.105367 15304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:40.111390 15304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33945
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346694975122-12919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:40.112581 15304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:40.113747 15304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:40.122968 15311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:40.122977 15316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:40.123080 15304 server_base.cc:1034] running on GCE node
W20260810 07:29:40.122962 15313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:40.123914 15304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:40.125206 15304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:40.126405 15304 hybrid_clock.cc:648] HybridClock initialized: now 1786346980126385 us; error 34 us; skew 500 ppm
Aug 10 07:29:40 dist-test-slave-ffrd krb5kdc[15283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:40.132143 15304 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 07:29:40.133358 15304 webserver.cc:460] Webserver started at http://127.0.0.1:43883/ using document root <none> and password file <none>
I20260810 07:29:40.133932 15304 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:40.133996 15304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:40.134172 15304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:40.136245 15304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "6a10110cc2b64be2a5ebfb815cb9565b"
format_stamp: "Formatted at 2026-08-10 07:29:40 on dist-test-slave-ffrd"
server_key: "b25576575a9394d77165e38301b6107d"
server_key_iv: "9f8128f4188766aa66a0949563956cf2"
server_key_version: "encryptionkey@0"
I20260810 07:29:40.136854 15304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "6a10110cc2b64be2a5ebfb815cb9565b"
format_stamp: "Formatted at 2026-08-10 07:29:40 on dist-test-slave-ffrd"
server_key: "b25576575a9394d77165e38301b6107d"
server_key_iv: "9f8128f4188766aa66a0949563956cf2"
server_key_version: "encryptionkey@0"
I20260810 07:29:40.140595 15304 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 07:29:40.143313 15327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:40.144902 15304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 07:29:40.145116 15304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "6a10110cc2b64be2a5ebfb815cb9565b"
format_stamp: "Formatted at 2026-08-10 07:29:40 on dist-test-slave-ffrd"
server_key: "b25576575a9394d77165e38301b6107d"
server_key_iv: "9f8128f4188766aa66a0949563956cf2"
server_key_version: "encryptionkey@0"
I20260810 07:29:40.145244 15304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:40.150130 15304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:40.153535 15304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:40.153748 15304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:40.161478 15304 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33945
I20260810 07:29:40.161576 15414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33945 every 8 connection(s)
I20260810 07:29:40.162559 15304 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:40.165467 15417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:40.171144 15417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a10110cc2b64be2a5ebfb815cb9565b: Bootstrap starting.
I20260810 07:29:40.172266 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 15304
I20260810 07:29:40.172452 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:29:40.172714 12919 external_mini_cluster.cc:1428] Setting key 987f5c7d70b9befd5b4fc9a92b9c3a57
I20260810 07:29:40.174319 15417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a10110cc2b64be2a5ebfb815cb9565b: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:40.175156 15417 log.cc:824] T 00000000000000000000000000000000 P 6a10110cc2b64be2a5ebfb815cb9565b: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:40.177506 15417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a10110cc2b64be2a5ebfb815cb9565b: No bootstrap required, opened a new log
I20260810 07:29:40.178597 15420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:40.173952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:45917 (local address 127.0.0.1:33945)
0810 07:29:40.174677 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:40.174713 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:40.174754 (+ 41us) server_negotiation.cc:407] Connection header received
0810 07:29:40.175611 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:40.175640 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:40.176028 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:40.176412 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:40.177301 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:40.177306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:40.177317 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:40.177329 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:40.177386 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:40.177406 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:40.177552 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:40.177663 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 07:29:40.177843 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":151,"threads_started":1}
I20260810 07:29:40.181821 12919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:29:40.182207 15417 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a10110cc2b64be2a5ebfb815cb9565b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a10110cc2b64be2a5ebfb815cb9565b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33945 } }
I20260810 07:29:40.182471 15417 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a10110cc2b64be2a5ebfb815cb9565b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:40.182547 15417 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a10110cc2b64be2a5ebfb815cb9565b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a10110cc2b64be2a5ebfb815cb9565b, State: Initialized, Role: FOLLOWER
I20260810 07:29:40.182986 15417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a10110cc2b64be2a5ebfb815cb9565b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a10110cc2b64be2a5ebfb815cb9565b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33945 } }
I20260810 07:29:40.183144 15417 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a10110cc2b64be2a5ebfb815cb9565b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:29:40.183215 15417 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a10110cc2b64be2a5ebfb815cb9565b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:29:40.183332 15417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a10110cc2b64be2a5ebfb815cb9565b [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:40.184517 15417 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a10110cc2b64be2a5ebfb815cb9565b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a10110cc2b64be2a5ebfb815cb9565b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33945 } }
I20260810 07:29:40.184929 15417 leader_election.cc:302] T 00000000000000000000000000000000 P 6a10110cc2b64be2a5ebfb815cb9565b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a10110cc2b64be2a5ebfb815cb9565b; no voters:
I20260810 07:29:40.185360 15417 leader_election.cc:288] T 00000000000000000000000000000000 P 6a10110cc2b64be2a5ebfb815cb9565b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:29:40.185479 15431 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a10110cc2b64be2a5ebfb815cb9565b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:40.185717 15431 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a10110cc2b64be2a5ebfb815cb9565b [term 1 LEADER]: Becoming Leader. State: Replica: 6a10110cc2b64be2a5ebfb815cb9565b, State: Running, Role: LEADER
I20260810 07:29:40.186239 15431 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a10110cc2b64be2a5ebfb815cb9565b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a10110cc2b64be2a5ebfb815cb9565b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33945 } }
I20260810 07:29:40.186286 15420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:40.184221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.105:50907 (local address 127.0.0.1:33945)
0810 07:29:40.184366 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:40.184374 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:40.184507 (+ 133us) server_negotiation.cc:407] Connection header received
0810 07:29:40.184586 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:40.184589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:40.184633 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:40.184715 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:40.185673 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:40.185677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:40.185679 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:40.185692 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:40.185723 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:40.185736 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:40.186099 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:40.186133 (+ 34us) server_negotiation.cc:299] Negotiation successful
0810 07:29:40.186174 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 07:29:40.186861 15417 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a10110cc2b64be2a5ebfb815cb9565b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:40.187574 15431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a10110cc2b64be2a5ebfb815cb9565b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a10110cc2b64be2a5ebfb815cb9565b. Latest consensus state: current_term: 1 leader_uuid: "6a10110cc2b64be2a5ebfb815cb9565b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a10110cc2b64be2a5ebfb815cb9565b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33945 } } }
I20260810 07:29:40.187706 15431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a10110cc2b64be2a5ebfb815cb9565b [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:40.188225 15432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a10110cc2b64be2a5ebfb815cb9565b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a10110cc2b64be2a5ebfb815cb9565b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a10110cc2b64be2a5ebfb815cb9565b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33945 } } }
I20260810 07:29:40.188377 15432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a10110cc2b64be2a5ebfb815cb9565b [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:40.188829 15439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:40.192559 15439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:40.198609 15439 catalog_manager.cc:1347] Generated new cluster ID: b3a1ce65378143f8b8223efbd258991b
I20260810 07:29:40.198807 15439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:40.218194 15439 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:40.218571 15439 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:40.232719 15439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a10110cc2b64be2a5ebfb815cb9565b: Generated new TSK 0
I20260810 07:29:40.233877 15439 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:29:40.244701 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 15304
2026-08-10T07:29:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1681 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:29:40 dist-test-slave-ffrd krb5kdc[15463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:29:40 dist-test-slave-ffrd krb5kdc[15463](info): set up 2 sockets
Aug 10 07:29:40 dist-test-slave-ffrd krb5kdc[15463](info): commencing operation
krb5kdc: starting...
W20260810 07:29:41.532820 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.261s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:29:41 dist-test-slave-ffrd krb5kdc[15463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:29:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:29:41.672742 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:41743
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:36119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:41743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:41.787094 15487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:41.787407 15487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:41.787453 15487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:41.792548 15487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:41.792656 15487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:41.792680 15487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:41.792701 15487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:41.792789 15487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:41.799153 15487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:36119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:41743
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:41743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:41.800488 15487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:41.801610 15487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:41.811532 15495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:41.811539 15496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:41.811583 15499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:41.812040 15487 server_base.cc:1034] running on GCE node
I20260810 07:29:41.812690 15487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:41.813951 15487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:41.815308 15487 hybrid_clock.cc:648] HybridClock initialized: now 1786346981815287 us; error 51 us; skew 500 ppm
Aug 10 07:29:41 dist-test-slave-ffrd krb5kdc[15463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346981, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:41.818783 15487 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:29:41.819717 15487 webserver.cc:460] Webserver started at http://127.12.157.254:32847/ using document root <none> and password file <none>
I20260810 07:29:41.820269 15487 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:41.820333 15487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:41.820514 15487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:41.823031 15487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "531d38cb31224692b7e324514fe99f5c"
format_stamp: "Formatted at 2026-08-10 07:29:41 on dist-test-slave-ffrd"
server_key: "dc3f30d04d471188116bf43bc69bc53f"
server_key_iv: "3b7925b55dcfa5b1192e0b24ca04eae6"
server_key_version: "encryptionkey@0"
I20260810 07:29:41.823586 15487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "531d38cb31224692b7e324514fe99f5c"
format_stamp: "Formatted at 2026-08-10 07:29:41 on dist-test-slave-ffrd"
server_key: "dc3f30d04d471188116bf43bc69bc53f"
server_key_iv: "3b7925b55dcfa5b1192e0b24ca04eae6"
server_key_version: "encryptionkey@0"
I20260810 07:29:41.827673 15487 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260810 07:29:41.830417 15507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:41.831816 15487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 07:29:41.831936 15487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "531d38cb31224692b7e324514fe99f5c"
format_stamp: "Formatted at 2026-08-10 07:29:41 on dist-test-slave-ffrd"
server_key: "dc3f30d04d471188116bf43bc69bc53f"
server_key_iv: "3b7925b55dcfa5b1192e0b24ca04eae6"
server_key_version: "encryptionkey@0"
I20260810 07:29:41.832031 15487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:41.846882 15487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:41.850277 15487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:41.850450 15487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:41.858072 15487 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:41743
I20260810 07:29:41.858158 15584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:41743 every 8 connection(s)
I20260810 07:29:41.859122 15487 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:41.861176 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 15487
I20260810 07:29:41.861318 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:29:41.861619 12919 external_mini_cluster.cc:1428] Setting key f6151afa676d3ba23b41de11ecb1ef15
I20260810 07:29:41.862426 15585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:41.869298 15585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 531d38cb31224692b7e324514fe99f5c: Bootstrap starting.
Aug 10 07:29:41 dist-test-slave-ffrd krb5kdc[15463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:29:41.872412 15585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 531d38cb31224692b7e324514fe99f5c: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:41.873453 15585 log.cc:824] T 00000000000000000000000000000000 P 531d38cb31224692b7e324514fe99f5c: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:41.875952 15585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 531d38cb31224692b7e324514fe99f5c: No bootstrap required, opened a new log
I20260810 07:29:41.878147 15589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:41.862998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43035 (local address 127.12.157.254:41743)
0810 07:29:41.863429 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:41.863452 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:41.863490 (+ 38us) server_negotiation.cc:407] Connection header received
0810 07:29:41.864132 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:41.864148 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:41.864473 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:41.864749 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:41.865726 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:41.866935 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:41.868210 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:41.868720 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:41.868870 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:41.872709 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:41.872717 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:41.872750 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:41.872793 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:41.874923 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:41.875544 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:41.875549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:41.875554 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:41.875669 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:41.876083 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:41.876086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:41.876088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:41.876523 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:41.876699 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:41.877124 (+ 425us) server_negotiation.cc:299] Negotiation successful
0810 07:29:41.877413 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":150,"threads_started":1}
I20260810 07:29:41.880049 15585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 531d38cb31224692b7e324514fe99f5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531d38cb31224692b7e324514fe99f5c" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41743 } }
I20260810 07:29:41.880321 15585 raft_consensus.cc:374] T 00000000000000000000000000000000 P 531d38cb31224692b7e324514fe99f5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:41.880384 15585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 531d38cb31224692b7e324514fe99f5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 531d38cb31224692b7e324514fe99f5c, State: Initialized, Role: FOLLOWER
I20260810 07:29:41.880883 15585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 531d38cb31224692b7e324514fe99f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531d38cb31224692b7e324514fe99f5c" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41743 } }
I20260810 07:29:41.881071 15585 raft_consensus.cc:388] T 00000000000000000000000000000000 P 531d38cb31224692b7e324514fe99f5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:29:41.881143 15585 raft_consensus.cc:482] T 00000000000000000000000000000000 P 531d38cb31224692b7e324514fe99f5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:29:41.881270 15585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 531d38cb31224692b7e324514fe99f5c [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:41.881433 12919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:29:41.882536 15585 raft_consensus.cc:504] T 00000000000000000000000000000000 P 531d38cb31224692b7e324514fe99f5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531d38cb31224692b7e324514fe99f5c" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41743 } }
I20260810 07:29:41.882941 15585 leader_election.cc:302] T 00000000000000000000000000000000 P 531d38cb31224692b7e324514fe99f5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 531d38cb31224692b7e324514fe99f5c; no voters:
I20260810 07:29:41.883296 15585 leader_election.cc:288] T 00000000000000000000000000000000 P 531d38cb31224692b7e324514fe99f5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:29:41.883419 15595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 531d38cb31224692b7e324514fe99f5c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:41.883688 15595 raft_consensus.cc:686] T 00000000000000000000000000000000 P 531d38cb31224692b7e324514fe99f5c [term 1 LEADER]: Becoming Leader. State: Replica: 531d38cb31224692b7e324514fe99f5c, State: Running, Role: LEADER
I20260810 07:29:41.884109 15595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 531d38cb31224692b7e324514fe99f5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531d38cb31224692b7e324514fe99f5c" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41743 } }
I20260810 07:29:41.884692 15585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 531d38cb31224692b7e324514fe99f5c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:41.885984 15598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 531d38cb31224692b7e324514fe99f5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "531d38cb31224692b7e324514fe99f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531d38cb31224692b7e324514fe99f5c" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41743 } } }
I20260810 07:29:41.886124 15598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 531d38cb31224692b7e324514fe99f5c [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:41.886377 15598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 531d38cb31224692b7e324514fe99f5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 531d38cb31224692b7e324514fe99f5c. Latest consensus state: current_term: 1 leader_uuid: "531d38cb31224692b7e324514fe99f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531d38cb31224692b7e324514fe99f5c" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 41743 } } }
I20260810 07:29:41.886476 15598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 531d38cb31224692b7e324514fe99f5c [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:41.887122 15611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:41.891988 15611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:41.893848 15589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:41.883814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38627 (local address 127.12.157.254:41743)
0810 07:29:41.883956 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:41.883960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:41.884408 (+ 448us) server_negotiation.cc:407] Connection header received
0810 07:29:41.884520 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:41.884524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:41.884575 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:41.884667 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:41.885572 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:41.886415 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:41.887659 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:41.887924 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:41.888033 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:41.889747 (+ 1714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:41.889751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:41.889753 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:41.889777 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:41.891573 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:41.892232 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:41.892236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:41.892238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:41.892336 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:41.892750 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:41.892753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:41.892755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:41.893051 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:41.893233 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:41.893566 (+ 333us) server_negotiation.cc:299] Negotiation successful
0810 07:29:41.893678 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 07:29:41.897997 15611 catalog_manager.cc:1347] Generated new cluster ID: 33d5eef5127d4923ae9bf02994926ace
I20260810 07:29:41.898072 15611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:41.915872 15611 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:41.917269 15611 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:41.924091 15611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 531d38cb31224692b7e324514fe99f5c: Generated new TSK 0
I20260810 07:29:41.924923 15611 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:29:41.929257 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 15487
W20260810 07:29:41.946197 15457 connection.cc:535] client connection to 127.12.157.254:41743 recv error: Network error: failed to read from TLS socket (remote: 127.12.157.254:41743): Connection reset by peer (error 104)
2026-08-10T07:29:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1689 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:29:41 dist-test-slave-ffrd krb5kdc[15629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:29:41 dist-test-slave-ffrd krb5kdc[15629](info): set up 2 sockets
Aug 10 07:29:41 dist-test-slave-ffrd krb5kdc[15629](info): commencing operation
krb5kdc: starting...
W20260810 07:29:43.223481 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.263s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:29:43 dist-test-slave-ffrd krb5kdc[15629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:29:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:29:43.349388 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:37899
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:39569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:37899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:43.466732 15650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:43.467078 15650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:43.467200 15650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:43.471328 15650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:43.471438 15650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:43.471473 15650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:43.471516 15650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:43.471549 15650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:43.477766 15650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:39569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:37899
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:37899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:43.478951 15650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:43.480036 15650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:43.490072 15667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:43.490034 15660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:43.490387 15664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:43.490760 15650 server_base.cc:1034] running on GCE node
I20260810 07:29:43.491536 15650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:43.492923 15650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:43.494205 15650 hybrid_clock.cc:648] HybridClock initialized: now 1786346983494179 us; error 43 us; skew 500 ppm
Aug 10 07:29:43 dist-test-slave-ffrd krb5kdc[15629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346983, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:43.497013 15650 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:29:43.497969 15650 webserver.cc:460] Webserver started at http://127.12.157.254:33355/ using document root <none> and password file <none>
I20260810 07:29:43.498561 15650 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:43.498656 15650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:43.498872 15650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:43.500983 15650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "3015a1ca4e0644f3ab616e3ff2a05a07"
format_stamp: "Formatted at 2026-08-10 07:29:43 on dist-test-slave-ffrd"
server_key: "3132d6df06149cfcaffa7d19d15e6b60"
server_key_iv: "6af1ce775992f9a4841529a31080a96e"
server_key_version: "encryptionkey@0"
I20260810 07:29:43.501606 15650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "3015a1ca4e0644f3ab616e3ff2a05a07"
format_stamp: "Formatted at 2026-08-10 07:29:43 on dist-test-slave-ffrd"
server_key: "3132d6df06149cfcaffa7d19d15e6b60"
server_key_iv: "6af1ce775992f9a4841529a31080a96e"
server_key_version: "encryptionkey@0"
I20260810 07:29:43.505450 15650 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 07:29:43.507923 15676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:43.509303 15650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:29:43.509449 15650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "3015a1ca4e0644f3ab616e3ff2a05a07"
format_stamp: "Formatted at 2026-08-10 07:29:43 on dist-test-slave-ffrd"
server_key: "3132d6df06149cfcaffa7d19d15e6b60"
server_key_iv: "6af1ce775992f9a4841529a31080a96e"
server_key_version: "encryptionkey@0"
I20260810 07:29:43.509573 15650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:43.539525 15650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:43.542835 15650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:43.543077 15650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:43.552325 15650 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:37899
I20260810 07:29:43.552359 15757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:37899 every 8 connection(s)
I20260810 07:29:43.553458 15650 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:43.557300 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 15650
I20260810 07:29:43.557466 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:29:43.557755 12919 external_mini_cluster.cc:1428] Setting key 1b18fcf52c3eb6d685d05733fb74414a
I20260810 07:29:43.557916 15758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:43.565899 15758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3015a1ca4e0644f3ab616e3ff2a05a07: Bootstrap starting.
I20260810 07:29:43.568895 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:43.559377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42273 (local address 127.12.157.254:37899)
0810 07:29:43.559813 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:43.559833 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:43.559875 (+ 42us) server_negotiation.cc:407] Connection header received
0810 07:29:43.560892 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:43.560918 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:43.561504 (+ 586us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:43.561936 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:43.562377 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:43.563890 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:43.564857 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:43.565564 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:43.565728 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:43.566771 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:43.566778 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:43.566793 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:43.567330 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:43.567407 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:43.567443 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:43.567589 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:43.567806 (+ 217us) server_negotiation.cc:299] Negotiation successful
0810 07:29:43.567951 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":115,"threads_started":1}
I20260810 07:29:43.568994 15758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3015a1ca4e0644f3ab616e3ff2a05a07: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:43.569859 15758 log.cc:824] T 00000000000000000000000000000000 P 3015a1ca4e0644f3ab616e3ff2a05a07: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:43.571975 12919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:29:43.572751 15758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3015a1ca4e0644f3ab616e3ff2a05a07: No bootstrap required, opened a new log
I20260810 07:29:43.576501 15758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3015a1ca4e0644f3ab616e3ff2a05a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3015a1ca4e0644f3ab616e3ff2a05a07" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37899 } }
I20260810 07:29:43.576756 15758 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3015a1ca4e0644f3ab616e3ff2a05a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:43.576820 15758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3015a1ca4e0644f3ab616e3ff2a05a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3015a1ca4e0644f3ab616e3ff2a05a07, State: Initialized, Role: FOLLOWER
I20260810 07:29:43.577391 15758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3015a1ca4e0644f3ab616e3ff2a05a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3015a1ca4e0644f3ab616e3ff2a05a07" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37899 } }
I20260810 07:29:43.577571 15758 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3015a1ca4e0644f3ab616e3ff2a05a07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:29:43.577648 15758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3015a1ca4e0644f3ab616e3ff2a05a07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:29:43.577751 15758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3015a1ca4e0644f3ab616e3ff2a05a07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:43.578049 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:43.574166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36737 (local address 127.12.157.254:37899)
0810 07:29:43.574296 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:43.574302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:43.574503 (+ 201us) server_negotiation.cc:407] Connection header received
0810 07:29:43.574636 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:43.574639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:43.574686 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:43.574771 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:43.575037 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:43.575544 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:43.576343 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:43.576575 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:43.576683 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:43.577567 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:43.577571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:43.577574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:43.577586 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:43.577620 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:43.577631 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:43.577736 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:43.577868 (+ 132us) server_negotiation.cc:299] Negotiation successful
0810 07:29:43.577906 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 07:29:43.578817 15758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3015a1ca4e0644f3ab616e3ff2a05a07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3015a1ca4e0644f3ab616e3ff2a05a07" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37899 } }
I20260810 07:29:43.579201 15758 leader_election.cc:302] T 00000000000000000000000000000000 P 3015a1ca4e0644f3ab616e3ff2a05a07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3015a1ca4e0644f3ab616e3ff2a05a07; no voters:
I20260810 07:29:43.579470 15758 leader_election.cc:288] T 00000000000000000000000000000000 P 3015a1ca4e0644f3ab616e3ff2a05a07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:29:43.579599 15771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3015a1ca4e0644f3ab616e3ff2a05a07 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:43.579839 15771 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3015a1ca4e0644f3ab616e3ff2a05a07 [term 1 LEADER]: Becoming Leader. State: Replica: 3015a1ca4e0644f3ab616e3ff2a05a07, State: Running, Role: LEADER
I20260810 07:29:43.580181 15771 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3015a1ca4e0644f3ab616e3ff2a05a07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3015a1ca4e0644f3ab616e3ff2a05a07" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37899 } }
I20260810 07:29:43.580687 15758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3015a1ca4e0644f3ab616e3ff2a05a07 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:43.581916 15772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3015a1ca4e0644f3ab616e3ff2a05a07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3015a1ca4e0644f3ab616e3ff2a05a07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3015a1ca4e0644f3ab616e3ff2a05a07" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37899 } } }
I20260810 07:29:43.581904 15774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3015a1ca4e0644f3ab616e3ff2a05a07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3015a1ca4e0644f3ab616e3ff2a05a07. Latest consensus state: current_term: 1 leader_uuid: "3015a1ca4e0644f3ab616e3ff2a05a07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3015a1ca4e0644f3ab616e3ff2a05a07" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37899 } } }
I20260810 07:29:43.582068 15772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3015a1ca4e0644f3ab616e3ff2a05a07 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:43.582068 15774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3015a1ca4e0644f3ab616e3ff2a05a07 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:43.582587 15784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:43.585618 15784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:43.591351 15784 catalog_manager.cc:1347] Generated new cluster ID: 66dfecbdc38742a7b21c9de13ff8d234
I20260810 07:29:43.591429 15784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:43.611423 15784 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:43.612376 15784 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:43.618530 15784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3015a1ca4e0644f3ab616e3ff2a05a07: Generated new TSK 0
I20260810 07:29:43.619210 15784 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:29:43.626420 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 15650
W20260810 07:29:43.639566 15627 connection.cc:535] client connection to 127.12.157.254:37899 recv error: Network error: failed to read from TLS socket (remote: 127.12.157.254:37899): Connection reset by peer (error 104)
2026-08-10T07:29:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1693 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:29:43 dist-test-slave-ffrd krb5kdc[15801](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:29:43 dist-test-slave-ffrd krb5kdc[15801](info): set up 2 sockets
Aug 10 07:29:43 dist-test-slave-ffrd krb5kdc[15801](info): commencing operation
krb5kdc: starting...
W20260810 07:29:44.891712 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.237s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:29:44 dist-test-slave-ffrd krb5kdc[15801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:29:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:29:45.024324 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:45875
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:43085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:45875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:45.136312 15819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:45.136601 15819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:45.136644 15819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:45.140702 15819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:45.140836 15819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:45.140877 15819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:45.140915 15819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:45.140965 15819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:45.146395 15819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:43085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:45875
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:45875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:45.147575 15819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:45.148640 15819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 07:29:45.157711 15819 server_base.cc:1034] running on GCE node
W20260810 07:29:45.157657 15828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:45.157903 15831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:45.157658 15829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:45.158577 15819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:45.159708 15819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:45.160887 15819 hybrid_clock.cc:648] HybridClock initialized: now 1786346985160858 us; error 40 us; skew 500 ppm
Aug 10 07:29:45 dist-test-slave-ffrd krb5kdc[15801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346985, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:45.167757 15819 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:29:45.168759 15819 webserver.cc:460] Webserver started at http://127.12.157.254:37157/ using document root <none> and password file <none>
I20260810 07:29:45.169588 15819 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:45.169667 15819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:45.169924 15819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:45.172189 15819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "088d8173336b44319cbb578c5a2a11ac"
format_stamp: "Formatted at 2026-08-10 07:29:45 on dist-test-slave-ffrd"
server_key: "1e98eb1d589b0aaded4ff9bff16c79d5"
server_key_iv: "fc44d69b08a8671c859cb70070dd36f7"
server_key_version: "encryptionkey@0"
I20260810 07:29:45.172837 15819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "088d8173336b44319cbb578c5a2a11ac"
format_stamp: "Formatted at 2026-08-10 07:29:45 on dist-test-slave-ffrd"
server_key: "1e98eb1d589b0aaded4ff9bff16c79d5"
server_key_iv: "fc44d69b08a8671c859cb70070dd36f7"
server_key_version: "encryptionkey@0"
I20260810 07:29:45.176833 15819 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260810 07:29:45.179396 15845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:45.180675 15819 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 07:29:45.180810 15819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "088d8173336b44319cbb578c5a2a11ac"
format_stamp: "Formatted at 2026-08-10 07:29:45 on dist-test-slave-ffrd"
server_key: "1e98eb1d589b0aaded4ff9bff16c79d5"
server_key_iv: "fc44d69b08a8671c859cb70070dd36f7"
server_key_version: "encryptionkey@0"
I20260810 07:29:45.180923 15819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:45.185745 15819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:45.189055 15819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:45.189250 15819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:45.196789 15819 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:45875
I20260810 07:29:45.196836 15944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:45875 every 8 connection(s)
I20260810 07:29:45.197834 15819 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:45.200783 15945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:45.201869 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 15819
I20260810 07:29:45.202021 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:29:45.202431 12919 external_mini_cluster.cc:1428] Setting key 34b2c13772b12087c765d395db4653ff
I20260810 07:29:45.208117 15945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 088d8173336b44319cbb578c5a2a11ac: Bootstrap starting.
I20260810 07:29:45.208585 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:45.203980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37855 (local address 127.12.157.254:45875)
0810 07:29:45.204450 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:45.204470 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:45.204515 (+ 45us) server_negotiation.cc:407] Connection header received
0810 07:29:45.205412 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:45.205438 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:45.205806 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:45.206177 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:45.207090 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:45.207096 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:45.207110 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:45.207123 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:45.207191 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:45.207219 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:45.207372 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:45.207501 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 07:29:45.207658 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":116,"threads_started":1}
I20260810 07:29:45.210677 15945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 088d8173336b44319cbb578c5a2a11ac: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:45.211531 15945 log.cc:824] T 00000000000000000000000000000000 P 088d8173336b44319cbb578c5a2a11ac: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:45.211591 12919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:29:45.214107 15945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 088d8173336b44319cbb578c5a2a11ac: No bootstrap required, opened a new log
I20260810 07:29:45.216298 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:45.214169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38829 (local address 127.12.157.254:45875)
0810 07:29:45.214296 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:45.214300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:45.214578 (+ 278us) server_negotiation.cc:407] Connection header received
0810 07:29:45.214672 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:45.214675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:45.214713 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:45.214821 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:45.215790 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:45.215794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:45.215797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:45.215810 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:45.215843 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:45.215856 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:45.215959 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:45.216114 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 07:29:45.216159 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 07:29:45.218614 15945 raft_consensus.cc:348] T 00000000000000000000000000000000 P 088d8173336b44319cbb578c5a2a11ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088d8173336b44319cbb578c5a2a11ac" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 45875 } }
I20260810 07:29:45.218860 15945 raft_consensus.cc:374] T 00000000000000000000000000000000 P 088d8173336b44319cbb578c5a2a11ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:45.218926 15945 raft_consensus.cc:729] T 00000000000000000000000000000000 P 088d8173336b44319cbb578c5a2a11ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 088d8173336b44319cbb578c5a2a11ac, State: Initialized, Role: FOLLOWER
I20260810 07:29:45.219377 15945 consensus_queue.cc:260] T 00000000000000000000000000000000 P 088d8173336b44319cbb578c5a2a11ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088d8173336b44319cbb578c5a2a11ac" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 45875 } }
I20260810 07:29:45.219511 15945 raft_consensus.cc:388] T 00000000000000000000000000000000 P 088d8173336b44319cbb578c5a2a11ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:29:45.219596 15945 raft_consensus.cc:482] T 00000000000000000000000000000000 P 088d8173336b44319cbb578c5a2a11ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:29:45.219717 15945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 088d8173336b44319cbb578c5a2a11ac [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:45.220733 15945 raft_consensus.cc:504] T 00000000000000000000000000000000 P 088d8173336b44319cbb578c5a2a11ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088d8173336b44319cbb578c5a2a11ac" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 45875 } }
I20260810 07:29:45.221155 15945 leader_election.cc:302] T 00000000000000000000000000000000 P 088d8173336b44319cbb578c5a2a11ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 088d8173336b44319cbb578c5a2a11ac; no voters:
I20260810 07:29:45.221519 15945 leader_election.cc:288] T 00000000000000000000000000000000 P 088d8173336b44319cbb578c5a2a11ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:29:45.221684 15965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 088d8173336b44319cbb578c5a2a11ac [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:45.221932 15965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 088d8173336b44319cbb578c5a2a11ac [term 1 LEADER]: Becoming Leader. State: Replica: 088d8173336b44319cbb578c5a2a11ac, State: Running, Role: LEADER
I20260810 07:29:45.222558 15965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 088d8173336b44319cbb578c5a2a11ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088d8173336b44319cbb578c5a2a11ac" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 45875 } }
I20260810 07:29:45.222920 15945 sys_catalog.cc:564] T 00000000000000000000000000000000 P 088d8173336b44319cbb578c5a2a11ac [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:45.224226 15968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 088d8173336b44319cbb578c5a2a11ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 088d8173336b44319cbb578c5a2a11ac. Latest consensus state: current_term: 1 leader_uuid: "088d8173336b44319cbb578c5a2a11ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088d8173336b44319cbb578c5a2a11ac" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 45875 } } }
I20260810 07:29:45.224362 15968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 088d8173336b44319cbb578c5a2a11ac [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:45.224676 15966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 088d8173336b44319cbb578c5a2a11ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "088d8173336b44319cbb578c5a2a11ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "088d8173336b44319cbb578c5a2a11ac" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 45875 } } }
I20260810 07:29:45.224756 15966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 088d8173336b44319cbb578c5a2a11ac [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:45.225111 15982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:45.228641 15982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:45.235790 15982 catalog_manager.cc:1347] Generated new cluster ID: 2c8ba1f428954331b5e6c8f4b1fefe00
I20260810 07:29:45.235879 15982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:45.245496 15982 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:45.245899 15982 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:45.259450 15982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 088d8173336b44319cbb578c5a2a11ac: Generated new TSK 0
I20260810 07:29:45.260452 15982 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:29:45.266021 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 15819
2026-08-10T07:29:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1637 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:29:45 dist-test-slave-ffrd krb5kdc[15999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:29:45 dist-test-slave-ffrd krb5kdc[15999](info): set up 2 sockets
Aug 10 07:29:45 dist-test-slave-ffrd krb5kdc[15999](info): commencing operation
krb5kdc: starting...
W20260810 07:29:46.536764 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.245s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:29:46 dist-test-slave-ffrd krb5kdc[15999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:29:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:29:46.678287 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:42561
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:42145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:42561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:46.795836 16022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:46.796147 16022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:46.796231 16022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:46.800190 16022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:46.800300 16022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:46.800340 16022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:46.800403 16022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:46.800444 16022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:46.805722 16022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:42145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:42561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:42561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:46.806854 16022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:46.807900 16022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:46.816881 16035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:46.816859 16031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:46.816859 16032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:46.817071 16022 server_base.cc:1034] running on GCE node
I20260810 07:29:46.817754 16022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:46.818907 16022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:46.820087 16022 hybrid_clock.cc:648] HybridClock initialized: now 1786346986820040 us; error 60 us; skew 500 ppm
Aug 10 07:29:46 dist-test-slave-ffrd krb5kdc[15999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346986, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:46.823203 16022 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:29:46.824162 16022 webserver.cc:460] Webserver started at http://127.12.157.254:40761/ using document root <none> and password file <none>
I20260810 07:29:46.824724 16022 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:46.824790 16022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:46.825064 16022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:46.827123 16022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "52a428e25929410eb59cc966d5846ca4"
format_stamp: "Formatted at 2026-08-10 07:29:46 on dist-test-slave-ffrd"
server_key: "c1af7c8b5fc0a7bddabf4f8560dbe1cd"
server_key_iv: "b7b0b6fbfcbc50ef42924ab0549b7ecf"
server_key_version: "encryptionkey@0"
I20260810 07:29:46.827714 16022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "52a428e25929410eb59cc966d5846ca4"
format_stamp: "Formatted at 2026-08-10 07:29:46 on dist-test-slave-ffrd"
server_key: "c1af7c8b5fc0a7bddabf4f8560dbe1cd"
server_key_iv: "b7b0b6fbfcbc50ef42924ab0549b7ecf"
server_key_version: "encryptionkey@0"
I20260810 07:29:46.831557 16022 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 07:29:46.834034 16044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:46.835309 16022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:29:46.835475 16022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "52a428e25929410eb59cc966d5846ca4"
format_stamp: "Formatted at 2026-08-10 07:29:46 on dist-test-slave-ffrd"
server_key: "c1af7c8b5fc0a7bddabf4f8560dbe1cd"
server_key_iv: "b7b0b6fbfcbc50ef42924ab0549b7ecf"
server_key_version: "encryptionkey@0"
I20260810 07:29:46.835597 16022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:46.869316 16022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:46.872896 16022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:46.873168 16022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:46.881520 16022 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:42561
I20260810 07:29:46.881549 16123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:42561 every 8 connection(s)
I20260810 07:29:46.882563 16022 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:46.885675 16124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:46.886680 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 16022
I20260810 07:29:46.886876 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:29:46.887195 12919 external_mini_cluster.cc:1428] Setting key eb8556a175ea8d97f09565af4af1cbe7
I20260810 07:29:46.893280 16124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52a428e25929410eb59cc966d5846ca4: Bootstrap starting.
I20260810 07:29:46.896247 16124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52a428e25929410eb59cc966d5846ca4: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:46.897140 16124 log.cc:824] T 00000000000000000000000000000000 P 52a428e25929410eb59cc966d5846ca4: Log is configured to *not* fsync() on all Append() calls
Aug 10 07:29:46 dist-test-slave-ffrd krb5kdc[15999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:29:46.900149 16124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52a428e25929410eb59cc966d5846ca4: No bootstrap required, opened a new log
I20260810 07:29:46.906086 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:46.888816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43399 (local address 127.12.157.254:42561)
0810 07:29:46.889365 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:46.889388 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:46.889434 (+ 46us) server_negotiation.cc:407] Connection header received
0810 07:29:46.890303 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:46.890329 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:46.890785 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:46.891154 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:46.892001 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:46.893684 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:46.895175 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:46.895703 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:46.895972 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:46.899656 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:46.899661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:46.899677 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:46.899708 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:46.902252 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:46.902802 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:46.902809 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:46.902815 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:46.902989 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:46.903301 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:46.903306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:46.903308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:46.903959 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:46.904152 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:46.904500 (+ 348us) server_negotiation.cc:299] Negotiation successful
0810 07:29:46.904865 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":135,"threads_started":1}
I20260810 07:29:46.905929 16124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52a428e25929410eb59cc966d5846ca4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52a428e25929410eb59cc966d5846ca4" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 42561 } }
I20260810 07:29:46.906230 16124 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52a428e25929410eb59cc966d5846ca4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:46.906284 16124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52a428e25929410eb59cc966d5846ca4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a428e25929410eb59cc966d5846ca4, State: Initialized, Role: FOLLOWER
I20260810 07:29:46.906790 16124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52a428e25929410eb59cc966d5846ca4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52a428e25929410eb59cc966d5846ca4" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 42561 } }
I20260810 07:29:46.906930 16124 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52a428e25929410eb59cc966d5846ca4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:29:46.907015 16124 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52a428e25929410eb59cc966d5846ca4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:29:46.907141 16124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52a428e25929410eb59cc966d5846ca4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:46.908169 16124 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52a428e25929410eb59cc966d5846ca4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52a428e25929410eb59cc966d5846ca4" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 42561 } }
I20260810 07:29:46.908540 16124 leader_election.cc:302] T 00000000000000000000000000000000 P 52a428e25929410eb59cc966d5846ca4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52a428e25929410eb59cc966d5846ca4; no voters:
I20260810 07:29:46.908912 16124 leader_election.cc:288] T 00000000000000000000000000000000 P 52a428e25929410eb59cc966d5846ca4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:29:46.909077 16137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52a428e25929410eb59cc966d5846ca4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:46.909361 16137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52a428e25929410eb59cc966d5846ca4 [term 1 LEADER]: Becoming Leader. State: Replica: 52a428e25929410eb59cc966d5846ca4, State: Running, Role: LEADER
I20260810 07:29:46.909740 16137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52a428e25929410eb59cc966d5846ca4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52a428e25929410eb59cc966d5846ca4" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 42561 } }
I20260810 07:29:46.909929 12919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:29:46.910408 16124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52a428e25929410eb59cc966d5846ca4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:46.911480 16139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52a428e25929410eb59cc966d5846ca4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52a428e25929410eb59cc966d5846ca4. Latest consensus state: current_term: 1 leader_uuid: "52a428e25929410eb59cc966d5846ca4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52a428e25929410eb59cc966d5846ca4" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 42561 } } }
I20260810 07:29:46.911623 16139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52a428e25929410eb59cc966d5846ca4 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:46.912000 16138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52a428e25929410eb59cc966d5846ca4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52a428e25929410eb59cc966d5846ca4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52a428e25929410eb59cc966d5846ca4" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 42561 } } }
I20260810 07:29:46.912146 16138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52a428e25929410eb59cc966d5846ca4 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:46.913182 16157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:46.916637 16157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:46.921900 16133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:46.913013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60435 (local address 127.12.157.254:42561)
0810 07:29:46.913128 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:46.913132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:46.913640 (+ 508us) server_negotiation.cc:407] Connection header received
0810 07:29:46.913760 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:46.913762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:46.913817 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:46.913926 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:46.914627 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:46.915164 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:46.915989 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:46.916231 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:46.916351 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:46.917875 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:46.917879 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:46.917882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:46.917918 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:46.919601 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:46.920124 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:46.920130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:46.920133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:46.920239 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:46.920591 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:46.920595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:46.920597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:46.920854 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:46.921042 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:46.921667 (+ 625us) server_negotiation.cc:299] Negotiation successful
0810 07:29:46.921774 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 07:29:46.923358 16157 catalog_manager.cc:1347] Generated new cluster ID: e3705fdb53114dcbae9e4adbf2c15f1e
I20260810 07:29:46.923442 16157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:46.953770 16157 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:46.955138 16157 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:46.961812 16157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52a428e25929410eb59cc966d5846ca4: Generated new TSK 0
I20260810 07:29:46.962611 16157 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:29:46.969314 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 16022
2026-08-10T07:29:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1708 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:29:47 dist-test-slave-ffrd krb5kdc[16179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:29:47 dist-test-slave-ffrd krb5kdc[16179](info): set up 2 sockets
Aug 10 07:29:47 dist-test-slave-ffrd krb5kdc[16179](info): commencing operation
krb5kdc: starting...
W20260810 07:29:48.258795 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.259s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:29:48 dist-test-slave-ffrd krb5kdc[16179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:29:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:29:48.402570 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:36425
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:37575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:36425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:48.524787 16199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:48.525104 16199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:48.525148 16199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:48.529119 16199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:48.529184 16199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:48.529203 16199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:48.529220 16199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:48.529235 16199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:48.534701 16199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:37575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:36425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:36425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:48.535871 16199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:48.536907 16199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:48.545976 16205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:48.546012 16204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:48.545964 16209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:48.546597 16199 server_base.cc:1034] running on GCE node
I20260810 07:29:48.547199 16199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:48.549043 16199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:48.550256 16199 hybrid_clock.cc:648] HybridClock initialized: now 1786346988550149 us; error 131 us; skew 500 ppm
Aug 10 07:29:48 dist-test-slave-ffrd krb5kdc[16179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346988, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:48.553651 16199 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:29:48.554656 16199 webserver.cc:460] Webserver started at http://127.12.157.254:33855/ using document root <none> and password file <none>
I20260810 07:29:48.555220 16199 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:48.555282 16199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:48.555459 16199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:48.557719 16199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "4b079996d87b477380fca99c35bb5422"
format_stamp: "Formatted at 2026-08-10 07:29:48 on dist-test-slave-ffrd"
server_key: "2acd335e1db445222913444da5e18a31"
server_key_iv: "df6e725f0dc435ffc71759ebaeba29bf"
server_key_version: "encryptionkey@0"
I20260810 07:29:48.558275 16199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "4b079996d87b477380fca99c35bb5422"
format_stamp: "Formatted at 2026-08-10 07:29:48 on dist-test-slave-ffrd"
server_key: "2acd335e1db445222913444da5e18a31"
server_key_iv: "df6e725f0dc435ffc71759ebaeba29bf"
server_key_version: "encryptionkey@0"
I20260810 07:29:48.562386 16199 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 07:29:48.565038 16218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:48.566473 16199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 07:29:48.566586 16199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "4b079996d87b477380fca99c35bb5422"
format_stamp: "Formatted at 2026-08-10 07:29:48 on dist-test-slave-ffrd"
server_key: "2acd335e1db445222913444da5e18a31"
server_key_iv: "df6e725f0dc435ffc71759ebaeba29bf"
server_key_version: "encryptionkey@0"
I20260810 07:29:48.566682 16199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:48.584425 16199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:48.587778 16199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:48.587952 16199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:48.596241 16199 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:36425
I20260810 07:29:48.596256 16297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:36425 every 8 connection(s)
I20260810 07:29:48.597303 16199 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:48.600539 16299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:48.602516 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 16199
I20260810 07:29:48.602658 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:29:48.602952 12919 external_mini_cluster.cc:1428] Setting key 00e71974379e6f0803396e678fcba01b
I20260810 07:29:48.607481 16299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b079996d87b477380fca99c35bb5422: Bootstrap starting.
I20260810 07:29:48.609869 16299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b079996d87b477380fca99c35bb5422: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:48.610620 16299 log.cc:824] T 00000000000000000000000000000000 P 4b079996d87b477380fca99c35bb5422: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:48.612742 16303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:48.604417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33325 (local address 127.12.157.254:36425)
0810 07:29:48.604873 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:48.604888 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:48.604925 (+ 37us) server_negotiation.cc:407] Connection header received
0810 07:29:48.605872 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:48.605894 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:48.606259 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:48.606686 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:48.607112 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:48.608146 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:48.609250 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:48.609749 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:48.609934 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:48.610813 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:48.610818 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:48.610832 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:48.611435 (+ 603us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:48.611512 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:48.611546 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:48.611675 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:48.611793 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 07:29:48.612018 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":140,"threads_started":1}
I20260810 07:29:48.613132 16299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b079996d87b477380fca99c35bb5422: No bootstrap required, opened a new log
I20260810 07:29:48.616039 12919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:29:48.617764 16299 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b079996d87b477380fca99c35bb5422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b079996d87b477380fca99c35bb5422" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 36425 } }
I20260810 07:29:48.618062 16299 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b079996d87b477380fca99c35bb5422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:48.618141 16299 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b079996d87b477380fca99c35bb5422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b079996d87b477380fca99c35bb5422, State: Initialized, Role: FOLLOWER
I20260810 07:29:48.618733 16299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b079996d87b477380fca99c35bb5422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b079996d87b477380fca99c35bb5422" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 36425 } }
I20260810 07:29:48.618956 16299 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b079996d87b477380fca99c35bb5422 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:29:48.619037 16299 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b079996d87b477380fca99c35bb5422 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:29:48.619145 16299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b079996d87b477380fca99c35bb5422 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:48.620388 16299 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b079996d87b477380fca99c35bb5422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b079996d87b477380fca99c35bb5422" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 36425 } }
I20260810 07:29:48.620750 16299 leader_election.cc:302] T 00000000000000000000000000000000 P 4b079996d87b477380fca99c35bb5422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b079996d87b477380fca99c35bb5422; no voters:
I20260810 07:29:48.621155 16299 leader_election.cc:288] T 00000000000000000000000000000000 P 4b079996d87b477380fca99c35bb5422 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:29:48.621269 16316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b079996d87b477380fca99c35bb5422 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:48.621507 16316 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b079996d87b477380fca99c35bb5422 [term 1 LEADER]: Becoming Leader. State: Replica: 4b079996d87b477380fca99c35bb5422, State: Running, Role: LEADER
I20260810 07:29:48.621822 16316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b079996d87b477380fca99c35bb5422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b079996d87b477380fca99c35bb5422" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 36425 } }
I20260810 07:29:48.622606 16299 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b079996d87b477380fca99c35bb5422 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:48.623169 16303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:48.618339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33141 (local address 127.12.157.254:36425)
0810 07:29:48.618483 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:48.618488 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:48.618725 (+ 237us) server_negotiation.cc:407] Connection header received
0810 07:29:48.618851 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:48.618853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:48.618921 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:48.619032 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:48.619330 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:48.620099 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:48.621045 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:48.621441 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:48.621544 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:48.622510 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:48.622514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:48.622517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:48.622531 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:48.622563 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:48.622576 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:48.622655 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:48.622973 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 07:29:48.623037 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 07:29:48.624046 16318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b079996d87b477380fca99c35bb5422 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b079996d87b477380fca99c35bb5422. Latest consensus state: current_term: 1 leader_uuid: "4b079996d87b477380fca99c35bb5422" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b079996d87b477380fca99c35bb5422" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 36425 } } }
I20260810 07:29:48.624179 16318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b079996d87b477380fca99c35bb5422 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:48.624521 16317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b079996d87b477380fca99c35bb5422 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b079996d87b477380fca99c35bb5422" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b079996d87b477380fca99c35bb5422" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 36425 } } }
I20260810 07:29:48.624629 16317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b079996d87b477380fca99c35bb5422 [sys.catalog]: This master's current role is: LEADER
W20260810 07:29:48.628093 16338 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b079996d87b477380fca99c35bb5422: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:29:48.628180 16338 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:29:48.628268 16330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:48.629196 16330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:48.635632 16330 catalog_manager.cc:1347] Generated new cluster ID: cb292b132c5f42e287d2e3ee59940cb0
I20260810 07:29:48.635715 16330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:48.645463 16330 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:48.646492 16330 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:48.657845 16330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b079996d87b477380fca99c35bb5422: Generated new TSK 0
I20260810 07:29:48.658644 16330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:29:48.665674 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 16199
2026-08-10T07:29:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1690 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:29:48 dist-test-slave-ffrd krb5kdc[16349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:29:48 dist-test-slave-ffrd krb5kdc[16349](info): set up 2 sockets
Aug 10 07:29:48 dist-test-slave-ffrd krb5kdc[16349](info): commencing operation
krb5kdc: starting...
W20260810 07:29:49.945791 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.254s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:29:50 dist-test-slave-ffrd krb5kdc[16349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:29:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:29:50.073892 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:33249
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:39945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:33249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:50.187079 16366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:50.187337 16366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:50.187378 16366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:50.191362 16366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:50.191422 16366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:50.191442 16366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:50.191457 16366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:50.191473 16366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:50.196683 16366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:39945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:33249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:33249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:50.197781 16366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:50.198817 16366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:50.207459 16376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:50.207494 16375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:50.207491 16378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:50.207696 16366 server_base.cc:1034] running on GCE node
I20260810 07:29:50.208312 16366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:50.209522 16366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:50.210712 16366 hybrid_clock.cc:648] HybridClock initialized: now 1786346990210677 us; error 51 us; skew 500 ppm
Aug 10 07:29:50 dist-test-slave-ffrd krb5kdc[16349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346990, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:50.217389 16366 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:29:50.218353 16366 webserver.cc:460] Webserver started at http://127.12.157.254:40499/ using document root <none> and password file <none>
I20260810 07:29:50.218948 16366 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:50.219044 16366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:50.219296 16366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:50.221518 16366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "86c39dbddd0d4b719cb4849c2d6c2c99"
format_stamp: "Formatted at 2026-08-10 07:29:50 on dist-test-slave-ffrd"
server_key: "74ab232cf896ac7fa90e39f628feffc4"
server_key_iv: "f734d985caf855ae878fea542a356863"
server_key_version: "encryptionkey@0"
I20260810 07:29:50.222083 16366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "86c39dbddd0d4b719cb4849c2d6c2c99"
format_stamp: "Formatted at 2026-08-10 07:29:50 on dist-test-slave-ffrd"
server_key: "74ab232cf896ac7fa90e39f628feffc4"
server_key_iv: "f734d985caf855ae878fea542a356863"
server_key_version: "encryptionkey@0"
I20260810 07:29:50.225880 16366 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 07:29:50.228348 16386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:50.229650 16366 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 07:29:50.229807 16366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "86c39dbddd0d4b719cb4849c2d6c2c99"
format_stamp: "Formatted at 2026-08-10 07:29:50 on dist-test-slave-ffrd"
server_key: "74ab232cf896ac7fa90e39f628feffc4"
server_key_iv: "f734d985caf855ae878fea542a356863"
server_key_version: "encryptionkey@0"
I20260810 07:29:50.229931 16366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:50.234661 16366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:50.237620 16366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:50.237823 16366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:50.245565 16366 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:33249
I20260810 07:29:50.245653 16469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:33249 every 8 connection(s)
I20260810 07:29:50.246599 16366 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:50.249617 16470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:50.250566 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 16366
I20260810 07:29:50.250726 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:29:50.251031 12919 external_mini_cluster.cc:1428] Setting key 5e810906d2bc8655832413dc02d4d5ee
I20260810 07:29:50.256896 16470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86c39dbddd0d4b719cb4849c2d6c2c99: Bootstrap starting.
I20260810 07:29:50.257019 16474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:50.252508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55467 (local address 127.12.157.254:33249)
0810 07:29:50.252994 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:50.253016 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:50.253069 (+ 53us) server_negotiation.cc:407] Connection header received
0810 07:29:50.253944 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:50.253969 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:50.254331 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:50.254710 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:50.255477 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:50.255483 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:50.255498 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:50.255514 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:50.255586 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:50.255616 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:50.255781 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:50.255879 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 07:29:50.256050 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":114,"threads_started":1}
I20260810 07:29:50.259696 16470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86c39dbddd0d4b719cb4849c2d6c2c99: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:50.260212 12919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:29:50.260602 16470 log.cc:824] T 00000000000000000000000000000000 P 86c39dbddd0d4b719cb4849c2d6c2c99: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:50.263013 16470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86c39dbddd0d4b719cb4849c2d6c2c99: No bootstrap required, opened a new log
I20260810 07:29:50.264745 16474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:50.262809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42467 (local address 127.12.157.254:33249)
0810 07:29:50.262984 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:50.262988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:50.263250 (+ 262us) server_negotiation.cc:407] Connection header received
0810 07:29:50.263356 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:50.263359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:50.263403 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:50.263502 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:50.264246 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:50.264250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:50.264253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 07:29:50.264264 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:50.264295 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 07:29:50.264308 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:50.264402 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:50.264567 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 07:29:50.264615 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 07:29:50.267691 16470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86c39dbddd0d4b719cb4849c2d6c2c99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86c39dbddd0d4b719cb4849c2d6c2c99" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 33249 } }
I20260810 07:29:50.267938 16470 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86c39dbddd0d4b719cb4849c2d6c2c99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 07:29:50.267974 16470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86c39dbddd0d4b719cb4849c2d6c2c99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86c39dbddd0d4b719cb4849c2d6c2c99, State: Initialized, Role: FOLLOWER
I20260810 07:29:50.268419 16470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86c39dbddd0d4b719cb4849c2d6c2c99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86c39dbddd0d4b719cb4849c2d6c2c99" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 33249 } }
I20260810 07:29:50.268558 16470 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86c39dbddd0d4b719cb4849c2d6c2c99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 07:29:50.268601 16470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86c39dbddd0d4b719cb4849c2d6c2c99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 07:29:50.268677 16470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86c39dbddd0d4b719cb4849c2d6c2c99 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:50.269775 16470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86c39dbddd0d4b719cb4849c2d6c2c99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86c39dbddd0d4b719cb4849c2d6c2c99" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 33249 } }
I20260810 07:29:50.270100 16470 leader_election.cc:302] T 00000000000000000000000000000000 P 86c39dbddd0d4b719cb4849c2d6c2c99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86c39dbddd0d4b719cb4849c2d6c2c99; no voters:
I20260810 07:29:50.270473 16470 leader_election.cc:288] T 00000000000000000000000000000000 P 86c39dbddd0d4b719cb4849c2d6c2c99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 07:29:50.270603 16483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86c39dbddd0d4b719cb4849c2d6c2c99 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:50.270879 16483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86c39dbddd0d4b719cb4849c2d6c2c99 [term 1 LEADER]: Becoming Leader. State: Replica: 86c39dbddd0d4b719cb4849c2d6c2c99, State: Running, Role: LEADER
I20260810 07:29:50.271262 16483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86c39dbddd0d4b719cb4849c2d6c2c99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86c39dbddd0d4b719cb4849c2d6c2c99" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 33249 } }
I20260810 07:29:50.271910 16470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86c39dbddd0d4b719cb4849c2d6c2c99 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:50.273357 16485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86c39dbddd0d4b719cb4849c2d6c2c99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86c39dbddd0d4b719cb4849c2d6c2c99. Latest consensus state: current_term: 1 leader_uuid: "86c39dbddd0d4b719cb4849c2d6c2c99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86c39dbddd0d4b719cb4849c2d6c2c99" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 33249 } } }
I20260810 07:29:50.273492 16485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86c39dbddd0d4b719cb4849c2d6c2c99 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:50.273447 16484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86c39dbddd0d4b719cb4849c2d6c2c99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86c39dbddd0d4b719cb4849c2d6c2c99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86c39dbddd0d4b719cb4849c2d6c2c99" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 33249 } } }
I20260810 07:29:50.273562 16484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86c39dbddd0d4b719cb4849c2d6c2c99 [sys.catalog]: This master's current role is: LEADER
W20260810 07:29:50.277544 16503 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 86c39dbddd0d4b719cb4849c2d6c2c99: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:29:50.277642 16503 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:29:50.277729 16494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:50.278448 16494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:50.283953 16494 catalog_manager.cc:1347] Generated new cluster ID: b27873d0ef984cc2b1b77ec5d971629b
I20260810 07:29:50.284036 16494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:50.303889 16494 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:50.304258 16494 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:50.311678 16494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86c39dbddd0d4b719cb4849c2d6c2c99: Generated new TSK 0
I20260810 07:29:50.312527 16494 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 07:29:50.324214 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 16366
2026-08-10T07:29:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1664 ms)
[----------] 18 tests from AuthTokenIssuingTest (30362 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:29:50 dist-test-slave-ffrd krb5kdc[16514](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:29:50 dist-test-slave-ffrd krb5kdc[16514](info): set up 2 sockets
Aug 10 07:29:50 dist-test-slave-ffrd krb5kdc[16514](info): commencing operation
krb5kdc: starting...
W20260810 07:29:51.599318 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.244s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:29:51 dist-test-slave-ffrd krb5kdc[16514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:29:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:29:51.732522 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:46277
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:37499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:46277,127.12.157.253:46105,127.12.157.252:34291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:51.849186 16533 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 07:29:51.849515 16533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:51.849586 16533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:51.849660 16533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:51.853587 16533 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:51.853672 16533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:51.853710 16533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:51.853749 16533 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:51.853780 16533 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:51.859087 16533 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:37499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:46277,127.12.157.253:46105,127.12.157.252:34291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:46277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:51.860191 16533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:51.861285 16533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:51.869884 16541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:51.869932 16550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:51.869884 16540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:51.870157 16533 server_base.cc:1034] running on GCE node
I20260810 07:29:51.870908 16533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:51.871979 16533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:51.873158 16533 hybrid_clock.cc:648] HybridClock initialized: now 1786346991873133 us; error 38 us; skew 500 ppm
Aug 10 07:29:51 dist-test-slave-ffrd krb5kdc[16514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346991, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:51.875886 16533 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:29:51.876806 16533 webserver.cc:460] Webserver started at http://127.12.157.254:45093/ using document root <none> and password file <none>
I20260810 07:29:51.877425 16533 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:51.877513 16533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:51.877745 16533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:51.879858 16533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "307516bf5ae545dfaef76bfb6dba105e"
format_stamp: "Formatted at 2026-08-10 07:29:51 on dist-test-slave-ffrd"
server_key: "dd0f80534148449ca1c0aa8d0e2ccaf2"
server_key_iv: "d5c3815750a46771a4b12ea38a999bb9"
server_key_version: "encryptionkey@0"
I20260810 07:29:51.880440 16533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "307516bf5ae545dfaef76bfb6dba105e"
format_stamp: "Formatted at 2026-08-10 07:29:51 on dist-test-slave-ffrd"
server_key: "dd0f80534148449ca1c0aa8d0e2ccaf2"
server_key_iv: "d5c3815750a46771a4b12ea38a999bb9"
server_key_version: "encryptionkey@0"
I20260810 07:29:51.884225 16533 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 07:29:51.886775 16562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:51.887957 16533 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 07:29:51.888170 16533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "307516bf5ae545dfaef76bfb6dba105e"
format_stamp: "Formatted at 2026-08-10 07:29:51 on dist-test-slave-ffrd"
server_key: "dd0f80534148449ca1c0aa8d0e2ccaf2"
server_key_iv: "d5c3815750a46771a4b12ea38a999bb9"
server_key_version: "encryptionkey@0"
I20260810 07:29:51.888317 16533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:51.916761 16533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:51.920217 16533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:51.920536 16533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:51.928036 16533 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:46277
I20260810 07:29:51.928331 16654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:46277 every 8 connection(s)
I20260810 07:29:51.929121 16533 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:51.929956 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 16533
I20260810 07:29:51.930104 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:29:51.930559 12919 external_mini_cluster.cc:1428] Setting key f725aa796b626eb68bea80a72406e0d8
I20260810 07:29:51.932907 16656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:51.937064 16656 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46277 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.157.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.253@KRBTEST.COM" created.
Aug 10 07:29:51 dist-test-slave-ffrd krb5kdc[16514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346991, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:29:51.952565 16661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:51.940076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50112 (local address 127.12.157.254:46277)
0810 07:29:51.940331 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:51.940341 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:51.940374 (+ 33us) server_negotiation.cc:407] Connection header received
0810 07:29:51.941248 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:51.941253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:51.941449 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:51.941647 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:51.942710 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:51.943619 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:51.944672 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:51.945188 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:51.945368 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:51.947691 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:51.947697 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:51.947706 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:51.947727 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:51.949396 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:51.949874 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:51.949879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:51.949881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:51.949984 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:51.950295 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:51.950298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:51.950300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:51.950675 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:51.950818 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:51.951255 (+ 437us) server_negotiation.cc:299] Negotiation successful
0810 07:29:51.951535 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":159,"thread_start_us":93,"threads_started":1}
I20260810 07:29:51.952562 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:51.939301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46277 (local address 127.0.0.1:50112)
0810 07:29:51.939815 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:51.939858 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:51.940744 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:51.941699 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:51.941707 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:51.942123 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:51.942522 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:51.942546 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:51.943783 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:51.943789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:51.944546 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:51.944556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:51.945346 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:51.945350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:51.945466 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:51.946063 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:51.946079 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:51.947539 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:51.949521 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:51.949531 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:51.949540 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:51.949763 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:51.950081 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:51.950084 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:51.950086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:51.950196 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:51.950794 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:51.950800 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:51.950938 (+ 138us) client_negotiation.cc:769] Sending connection context
0810 07:29:51.951139 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 07:29:51.951564 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":139,"thread_start_us":125,"threads_started":1}
I20260810 07:29:51.955194 16656 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.253" port: 46105 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:29:51.955919 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:51.955673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:46105 (local address 127.0.0.1:45404)
0810 07:29:51.955783 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:51.955858 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.253:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 07:29:51.956149 16563 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.157.253:46105: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.253.keytab.
Entry for principal kudu/127.12.157.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.253.keytab.
I20260810 07:29:51.957082 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:51.956805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:46105 (local address 127.0.0.1:45420)
0810 07:29:51.956919 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:51.957000 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.253:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 07:29:51.957231 16656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.157.253:46105: Network error: Client connection negotiation failed: client connection to 127.12.157.253:46105: connect: Connection refused (error 111)
I20260810 07:29:51.957841 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.253:46105
--webserver_interface=127.12.157.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.253.keytab
--principal=kudu/127.12.157.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:37499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:46277,127.12.157.253:46105,127.12.157.252:34291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:29:52.006520 16656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.157.253" port: 46105 } attempt: 1
I20260810 07:29:52.007496 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.007130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:46105 (local address 127.0.0.1:45434)
0810 07:29:52.007324 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:52.007402 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.253:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 07:29:52.008255 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.008032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:46105 (local address 127.0.0.1:45448)
0810 07:29:52.008138 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:52.008190 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.253:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 07:29:52.008440 16656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.157.253:46105: Network error: Client connection negotiation failed: client connection to 127.12.157.253:46105: connect: Connection refused (error 111)
W20260810 07:29:52.074882 16668 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 07:29:52.075176 16668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:52.075228 16668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:52.075313 16668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 07:29:52.076624 16656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.157.253" port: 46105 } attempt: 2
I20260810 07:29:52.077570 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.077231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:46105 (local address 127.0.0.1:45452)
0810 07:29:52.077403 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:52.077474 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.253:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 07:29:52.078364 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.078135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:46105 (local address 127.0.0.1:45466)
0810 07:29:52.078244 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:52.078296 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.253:46105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 07:29:52.078526 16656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.157.253:46105: Network error: Client connection negotiation failed: client connection to 127.12.157.253:46105: connect: Connection refused (error 111)
W20260810 07:29:52.079955 16668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:52.080030 16668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:52.080076 16668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:52.080111 16668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:52.080147 16668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:52.085603 16668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:37499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:46277,127.12.157.253:46105,127.12.157.252:34291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.253.keytab
--principal=kudu/127.12.157.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.253:46105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.157.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:52.086674 16668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:52.087687 16668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:52.096189 16685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:52.096261 16681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:52.096221 16679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:52.096625 16668 server_base.cc:1034] running on GCE node
I20260810 07:29:52.097257 16668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:52.098435 16668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:52.099606 16668 hybrid_clock.cc:648] HybridClock initialized: now 1786346992099590 us; error 30 us; skew 500 ppm
Aug 10 07:29:52 dist-test-slave-ffrd krb5kdc[16514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346992, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:52.102414 16668 init.cc:377] Logged in from keytab as kudu/127.12.157.253@KRBTEST.COM (short username kudu)
I20260810 07:29:52.103247 16668 webserver.cc:460] Webserver started at http://127.12.157.253:36603/ using document root <none> and password file <none>
I20260810 07:29:52.103768 16668 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:52.103825 16668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:52.103999 16668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:52.106063 16668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-1/data/instance:
uuid: "275c970eda6046e190aff105bcb41dbe"
format_stamp: "Formatted at 2026-08-10 07:29:52 on dist-test-slave-ffrd"
server_key: "28b6c2868595c055af04feef307fda17"
server_key_iv: "5b41badd7400f60fa516ebb6acbc5584"
server_key_version: "encryptionkey@0"
I20260810 07:29:52.106578 16668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-1/wal/instance:
uuid: "275c970eda6046e190aff105bcb41dbe"
format_stamp: "Formatted at 2026-08-10 07:29:52 on dist-test-slave-ffrd"
server_key: "28b6c2868595c055af04feef307fda17"
server_key_iv: "5b41badd7400f60fa516ebb6acbc5584"
server_key_version: "encryptionkey@0"
I20260810 07:29:52.110216 16668 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 07:29:52.112496 16693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:52.113710 16668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:29:52.113848 16668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-1/wal
uuid: "275c970eda6046e190aff105bcb41dbe"
format_stamp: "Formatted at 2026-08-10 07:29:52 on dist-test-slave-ffrd"
server_key: "28b6c2868595c055af04feef307fda17"
server_key_iv: "5b41badd7400f60fa516ebb6acbc5584"
server_key_version: "encryptionkey@0"
I20260810 07:29:52.113967 16668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:52.128448 16668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:52.131614 16668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:52.131827 16668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:52.139403 16668 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.253:46105
I20260810 07:29:52.139688 16784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.253:46105 every 8 connection(s)
I20260810 07:29:52.140450 16668 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-1/data/info.pb
I20260810 07:29:52.143379 16785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:52.145119 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 16668
I20260810 07:29:52.145213 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-1/wal/instance
I20260810 07:29:52.145500 12919 external_mini_cluster.cc:1428] Setting key 029ce8acafbfea7f852ed4c51a55f03d
I20260810 07:29:52.147511 16785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46277 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.157.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.252@KRBTEST.COM" created.
Aug 10 07:29:52 dist-test-slave-ffrd krb5kdc[16514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346992, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.253@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:29:52.163209 16661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.150015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50124 (local address 127.12.157.254:46277)
0810 07:29:52.150210 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:52.150217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:52.151059 (+ 842us) server_negotiation.cc:407] Connection header received
0810 07:29:52.152063 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:52.152069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:52.152142 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:52.152267 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:52.153546 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.154431 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.155755 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.156068 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.156199 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:52.158965 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:52.158969 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:52.158971 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:52.158999 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:52.161085 (+ 2086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:52.161610 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:52.161614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:52.161616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:52.161699 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:52.162045 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:52.162049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:52.162051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:52.162242 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:52.162345 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:52.162898 (+ 553us) server_negotiation.cc:299] Negotiation successful
0810 07:29:52.163034 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 07:29:52.164053 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.150375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46277 (local address 127.0.0.1:50124)
0810 07:29:52.150886 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:52.150940 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:52.151842 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:52.152408 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:52.152415 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:52.152813 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:52.153376 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:52.153390 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.154576 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.154579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:52.155629 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:52.155641 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.156255 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.156260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:52.156457 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:52.157115 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:52.157132 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:52.158777 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:52.161205 (+ 2428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:52.161211 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:52.161228 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:52.161504 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:52.161792 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:52.161796 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:52.161798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:52.161949 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:52.162346 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:52.162355 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:52.162634 (+ 279us) client_negotiation.cc:769] Sending connection context
0810 07:29:52.162831 (+ 197us) client_negotiation.cc:240] Negotiation successful
0810 07:29:52.163193 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":156,"threads_started":1}
I20260810 07:29:52.165714 16785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.253" port: 46105 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:29:52.169701 16656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.157.253" port: 46105 } attempt: 3
Aug 10 07:29:52 dist-test-slave-ffrd krb5kdc[16514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346992, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.253@KRBTEST.COM for kudu/127.12.157.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.252.keytab.
Entry for principal kudu/127.12.157.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.252.keytab.
I20260810 07:29:52.174556 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.252:34291
--webserver_interface=127.12.157.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.252.keytab
--principal=kudu/127.12.157.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:37499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:46277,127.12.157.253:46105,127.12.157.252:34291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 07:29:52 dist-test-slave-ffrd krb5kdc[16514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346991, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for kudu/127.12.157.253@KRBTEST.COM
I20260810 07:29:52.177095 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.166390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:46105 (local address 127.0.0.1:45472)
0810 07:29:52.166508 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:52.166523 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:52.166609 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:52.167364 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:52.167367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:52.167512 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:52.167657 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:52.167663 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.168759 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.168764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:52.169362 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:52.169371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.170348 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.170355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:52.170497 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:52.171057 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:52.171075 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:52.172486 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:52.175015 (+ 2529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:52.175021 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:52.175026 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:52.175312 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:52.175649 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:52.175655 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:52.175659 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:52.175767 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:52.176484 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:52.176489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:52.176601 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 07:29:52.176724 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 07:29:52.176864 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 07:29:52.177220 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.166547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45472 (local address 127.12.157.253:46105)
0810 07:29:52.166931 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:52.166945 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:52.166974 (+ 29us) server_negotiation.cc:407] Connection header received
0810 07:29:52.167026 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:52.167035 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:52.167232 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:52.167353 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:52.167766 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.168600 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.169502 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.170037 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.170195 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:52.172656 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:52.172668 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:52.172684 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:52.172709 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:52.174896 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:52.175436 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:52.175442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:52.175444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:52.175541 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:52.175877 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:52.175880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:52.175882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:52.176267 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:52.176441 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:52.176812 (+ 371us) server_negotiation.cc:299] Negotiation successful
0810 07:29:52.177034 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":139,"threads_started":1}
I20260810 07:29:52.180363 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.170203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:46105 (local address 127.0.0.1:45486)
0810 07:29:52.170335 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:52.170364 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:52.170436 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:52.170832 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:52.170835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:52.171041 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:52.171197 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:52.171205 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.172162 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.172167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:52.172892 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:52.172902 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.173453 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.173457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:52.173552 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:52.174127 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:52.174143 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:52.175578 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:52.178592 (+ 3014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:52.178595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:52.178596 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:52.178922 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:52.179253 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:52.179256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:52.179258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:52.179357 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:52.179832 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:52.179835 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:52.179939 (+ 104us) client_negotiation.cc:769] Sending connection context
0810 07:29:52.180050 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 07:29:52.180194 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 07:29:52.180603 16796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.170227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45486 (local address 127.12.157.253:46105)
0810 07:29:52.170619 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:52.170628 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:52.170640 (+ 12us) server_negotiation.cc:407] Connection header received
0810 07:29:52.170675 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:52.170677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:52.170710 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:52.170814 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:52.171333 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.172052 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.173057 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.173333 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.173447 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:52.175726 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:52.175729 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:52.175731 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:52.175751 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:52.178458 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:52.179042 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:52.179048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:52.179052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:52.179149 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:52.179459 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:52.179465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:52.179470 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:52.179719 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:52.179832 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:52.180142 (+ 310us) server_negotiation.cc:299] Negotiation successful
0810 07:29:52.180319 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":185,"threads_started":1}
I20260810 07:29:52.180724 16785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.252" port: 34291 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:29:52.181499 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.181260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:34291 (local address 127.0.0.1:55916)
0810 07:29:52.181377 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:52.181444 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:34291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 07:29:52.181649 16656 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.252" port: 34291 } has no permanent_uuid. Determining permanent_uuid...
W20260810 07:29:52.181689 16696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.157.252:34291: connect: Connection refused (error 111)
I20260810 07:29:52.182528 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.182139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:34291 (local address 127.0.0.1:55918)
0810 07:29:52.182373 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:52.182435 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:34291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260810 07:29:52.183396 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.183060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:34291 (local address 127.0.0.1:55934)
0810 07:29:52.183233 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:52.183310 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:34291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260810 07:29:52.183604 16656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.157.252:34291: Network error: Client connection negotiation failed: client connection to 127.12.157.252:34291: connect: Connection refused (error 111)
I20260810 07:29:52.185729 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.185502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:34291 (local address 127.0.0.1:55950)
0810 07:29:52.185618 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:52.185671 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:34291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 07:29:52.185918 16785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.157.252:34291: Network error: Client connection negotiation failed: client connection to 127.12.157.252:34291: connect: Connection refused (error 111)
I20260810 07:29:52.214800 16656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.157.252" port: 34291 } attempt: 1
I20260810 07:29:52.215759 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.215387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:34291 (local address 127.0.0.1:55964)
0810 07:29:52.215579 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:52.215657 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:34291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 07:29:52.216504 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.216266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:34291 (local address 127.0.0.1:55974)
0810 07:29:52.216381 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:52.216433 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:34291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 07:29:52.216647 16656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.157.252:34291: Network error: Client connection negotiation failed: client connection to 127.12.157.252:34291: connect: Connection refused (error 111)
I20260810 07:29:52.235195 16785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.157.252" port: 34291 } attempt: 1
I20260810 07:29:52.236205 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.235820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:34291 (local address 127.0.0.1:55982)
0810 07:29:52.236031 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:52.236110 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:34291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 07:29:52.237028 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.236770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:34291 (local address 127.0.0.1:55996)
0810 07:29:52.236879 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:52.236933 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:34291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 07:29:52.237174 16785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.157.252:34291: Network error: Client connection negotiation failed: client connection to 127.12.157.252:34291: connect: Connection refused (error 111)
I20260810 07:29:52.291841 16656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.157.252" port: 34291 } attempt: 2
I20260810 07:29:52.292836 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.292423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:34291 (local address 127.0.0.1:55998)
0810 07:29:52.292657 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:52.292722 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:34291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260810 07:29:52.293639 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.293414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:34291 (local address 127.0.0.1:56008)
0810 07:29:52.293535 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:52.293585 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:34291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 07:29:52.293823 16656 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.157.252:34291: Network error: Client connection negotiation failed: client connection to 127.12.157.252:34291: connect: Connection refused (error 111)
W20260810 07:29:52.297737 16800 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 07:29:52.298022 16800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:52.298071 16800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:52.298166 16800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:52.302148 16800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:52.302199 16800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:52.302218 16800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:52.302234 16800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:52.302284 16800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:52.305321 16785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.157.252" port: 34291 } attempt: 2
I20260810 07:29:52.306033 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.305746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:34291 (local address 127.0.0.1:56018)
0810 07:29:52.305892 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:52.305953 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:34291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 07:29:52.306780 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.306568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:34291 (local address 127.0.0.1:56032)
0810 07:29:52.306670 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:52.306721 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:34291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 07:29:52.306933 16785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.157.252:34291: Network error: Client connection negotiation failed: client connection to 127.12.157.252:34291: connect: Connection refused (error 111)
I20260810 07:29:52.308125 16800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:37499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:46277,127.12.157.253:46105,127.12.157.252:34291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.252.keytab
--principal=kudu/127.12.157.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.252:34291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.157.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:52.309253 16800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:52.310277 16800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:52.318699 16813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:52.318681 16812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:52.318722 16820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:52.318928 16800 server_base.cc:1034] running on GCE node
I20260810 07:29:52.319512 16800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:52.320604 16800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:52.321801 16800 hybrid_clock.cc:648] HybridClock initialized: now 1786346992321780 us; error 28 us; skew 500 ppm
Aug 10 07:29:52 dist-test-slave-ffrd krb5kdc[16514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346992, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:52.324608 16800 init.cc:377] Logged in from keytab as kudu/127.12.157.252@KRBTEST.COM (short username kudu)
I20260810 07:29:52.325544 16800 webserver.cc:460] Webserver started at http://127.12.157.252:35923/ using document root <none> and password file <none>
I20260810 07:29:52.326110 16800 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:52.326200 16800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:52.326431 16800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:52.328445 16800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-2/data/instance:
uuid: "c127907788fd41949e62f0734353f674"
format_stamp: "Formatted at 2026-08-10 07:29:52 on dist-test-slave-ffrd"
server_key: "567fa8b4a4eb903ae9f1609c0b02f5c7"
server_key_iv: "166130d404c7a6e7067b80cdd7fa2b0f"
server_key_version: "encryptionkey@0"
I20260810 07:29:52.329077 16800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-2/wal/instance:
uuid: "c127907788fd41949e62f0734353f674"
format_stamp: "Formatted at 2026-08-10 07:29:52 on dist-test-slave-ffrd"
server_key: "567fa8b4a4eb903ae9f1609c0b02f5c7"
server_key_iv: "166130d404c7a6e7067b80cdd7fa2b0f"
server_key_version: "encryptionkey@0"
I20260810 07:29:52.332651 16800 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 07:29:52.335043 16831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:52.336189 16800 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 07:29:52.336328 16800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-2/wal
uuid: "c127907788fd41949e62f0734353f674"
format_stamp: "Formatted at 2026-08-10 07:29:52 on dist-test-slave-ffrd"
server_key: "567fa8b4a4eb903ae9f1609c0b02f5c7"
server_key_iv: "166130d404c7a6e7067b80cdd7fa2b0f"
server_key_version: "encryptionkey@0"
I20260810 07:29:52.336436 16800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:52.353905 16800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:52.357215 16800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:52.357456 16800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:52.364756 16800 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.252:34291
I20260810 07:29:52.365089 16913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.252:34291 every 8 connection(s)
I20260810 07:29:52.365787 16800 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-2/data/info.pb
I20260810 07:29:52.368862 16915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:52.373126 16915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46277 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:29:52.373888 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 16800
I20260810 07:29:52.374023 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786346694975122-12919-0/minicluster-data/master-2/wal/instance
I20260810 07:29:52.374271 12919 external_mini_cluster.cc:1428] Setting key 7c55829e8ec1ba10c3db4ab62128dfed
Aug 10 07:29:52 dist-test-slave-ffrd krb5kdc[16514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346992, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.252@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
Aug 10 07:29:52 dist-test-slave-ffrd krb5kdc[16514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.252@KRBTEST.COM
I20260810 07:29:52.387447 16661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.375363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50128 (local address 127.12.157.254:46277)
0810 07:29:52.375490 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:52.375494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:52.376383 (+ 889us) server_negotiation.cc:407] Connection header received
0810 07:29:52.377591 (+ 1208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:52.377594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:52.377649 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:52.377758 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:52.378691 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.379199 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.380305 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.380641 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.380721 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:52.383558 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:52.383562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:52.383564 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:52.383588 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:52.385339 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:52.385845 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:52.385849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:52.385850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:52.385938 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:52.386286 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:52.386290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:52.386291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:52.386496 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:52.386604 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:52.387132 (+ 528us) server_negotiation.cc:299] Negotiation successful
0810 07:29:52.387268 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 07:29:52.388315 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.375632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56036 (local address 127.12.157.252:34291)
0810 07:29:52.376191 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:52.376209 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:52.376251 (+ 42us) server_negotiation.cc:407] Connection header received
0810 07:29:52.377171 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:52.377195 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:52.377620 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:52.377815 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:52.378642 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.379616 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.380569 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.381158 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.381271 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:52.383921 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:52.383927 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:52.383937 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:52.383960 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:52.385866 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:52.386390 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:52.386396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:52.386397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:52.386481 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:52.386760 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:52.386763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:52.386765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:52.387009 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:52.387213 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:52.387443 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 07:29:52.387585 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":449,"spinlock_wait_cycles":7936,"thread_start_us":159,"threads_started":1}
I20260810 07:29:52.388315 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.375632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:46277 (local address 127.0.0.1:50128)
0810 07:29:52.376218 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:52.376267 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:52.377171 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:52.377798 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:52.377803 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:52.378258 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:52.378515 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:52.378529 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.379308 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.379311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:52.380173 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:52.380182 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.380782 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.380786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:52.381072 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:52.381756 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:52.381773 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:52.383330 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:52.385449 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:52.385459 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:52.385470 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:52.385734 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:52.386036 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:52.386039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:52.386041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:52.386184 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:52.386606 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:52.386616 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:52.386883 (+ 267us) client_negotiation.cc:769] Sending connection context
0810 07:29:52.387075 (+ 192us) client_negotiation.cc:240] Negotiation successful
0810 07:29:52.387402 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":30,"thread_start_us":155,"threads_started":1}
I20260810 07:29:52.389565 16915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.253" port: 46105 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:29:52.393009 16656 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.157.252" port: 34291 } attempt: 3
Aug 10 07:29:52 dist-test-slave-ffrd krb5kdc[16514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346992, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.252@KRBTEST.COM for kudu/127.12.157.253@KRBTEST.COM
Aug 10 07:29:52 dist-test-slave-ffrd krb5kdc[16514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
Aug 10 07:29:52 dist-test-slave-ffrd krb5kdc[16514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346991, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for kudu/127.12.157.252@KRBTEST.COM
I20260810 07:29:52.398070 16785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.157.252" port: 34291 } attempt: 3
I20260810 07:29:52.399526 16796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.390265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45498 (local address 127.12.157.253:46105)
0810 07:29:52.390412 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:52.390417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:52.390430 (+ 13us) server_negotiation.cc:407] Connection header received
0810 07:29:52.390474 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:52.390477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:52.390527 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:52.390608 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:52.391090 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.391683 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.392481 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.392757 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.392870 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:52.395230 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:52.395235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:52.395238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:52.395272 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:52.397516 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:52.398020 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:52.398026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:52.398028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:52.398120 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:52.398463 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:52.398467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:52.398472 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:52.398712 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:52.398805 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:52.399203 (+ 398us) server_negotiation.cc:299] Negotiation successful
0810 07:29:52.399345 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 07:29:52.400030 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.390083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:46105 (local address 127.0.0.1:45498)
0810 07:29:52.390207 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:52.390219 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:52.390309 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:52.390622 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:52.390624 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:52.390823 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:52.390961 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:52.390969 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.391794 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.391799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:52.392363 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:52.392371 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.392868 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.392872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:52.393008 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:52.393464 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:52.393475 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:52.395044 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:52.397612 (+ 2568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:52.397617 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:52.397620 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:52.397889 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:52.398246 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:52.398248 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:52.398250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:52.398346 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:52.398894 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:52.398899 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:52.398999 (+ 100us) client_negotiation.cc:769] Sending connection context
0810 07:29:52.399719 (+ 720us) client_negotiation.cc:240] Negotiation successful
0810 07:29:52.399872 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 07:29:52.400746 16661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.390955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50130 (local address 127.12.157.254:46277)
0810 07:29:52.391078 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:52.391082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:52.391095 (+ 13us) server_negotiation.cc:407] Connection header received
0810 07:29:52.391223 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:52.391226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:52.391268 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:52.391339 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:52.391922 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.392528 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.393511 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.393809 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.393907 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:52.396677 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:52.396680 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:52.396688 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:52.396710 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:52.398712 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:52.399199 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:52.399202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:52.399203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:52.399302 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:52.399608 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:52.399612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:52.399613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:52.399828 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:52.400054 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:52.400428 (+ 374us) server_negotiation.cc:299] Negotiation successful
0810 07:29:52.400572 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50,"spinlock_wait_cycles":4480}
I20260810 07:29:52.400769 16915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.252" port: 34291 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:29:52.402237 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.393711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:34291 (local address 127.0.0.1:56048)
0810 07:29:52.393824 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:52.393838 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:52.393904 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:52.394215 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:52.394218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:52.394425 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:52.394531 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:52.394540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.395254 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.395259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:52.395890 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:52.395899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.396401 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.396405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:52.396498 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:52.396941 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:52.396985 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:52.398383 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:52.400361 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:52.400364 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:52.400366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:52.400640 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:52.401142 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:52.401145 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:52.401147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:52.401240 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:52.401756 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:52.401761 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:52.401858 (+ 97us) client_negotiation.cc:769] Sending connection context
0810 07:29:52.401944 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 07:29:52.402082 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"spinlock_wait_cycles":1408}
I20260810 07:29:52.402835 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.393887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56048 (local address 127.12.157.252:34291)
0810 07:29:52.394017 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:52.394023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:52.394037 (+ 14us) server_negotiation.cc:407] Connection header received
0810 07:29:52.394068 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:52.394071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:52.394112 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:52.394198 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:52.394642 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.395147 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.396014 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.396291 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.396396 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:52.398542 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:52.398546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:52.398548 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:52.398572 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:52.400239 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:52.400887 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:52.400893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:52.400896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:52.401022 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:52.401389 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:52.401395 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:52.401400 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:52.401636 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:52.402445 (+ 809us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:52.402529 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 07:29:52.402683 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Aug 10 07:29:52 dist-test-slave-ffrd krb5kdc[16514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346992, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.253@KRBTEST.COM for kudu/127.12.157.252@KRBTEST.COM
I20260810 07:29:52.407078 16656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 307516bf5ae545dfaef76bfb6dba105e: Bootstrap starting.
Aug 10 07:29:52 dist-test-slave-ffrd krb5kdc[16514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346992, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.252@KRBTEST.COM for kudu/127.12.157.252@KRBTEST.COM
Aug 10 07:29:52 dist-test-slave-ffrd krb5kdc[16514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.253@KRBTEST.COM
I20260810 07:29:52.411484 16656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 307516bf5ae545dfaef76bfb6dba105e: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:52.411573 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.398495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:34291 (local address 127.0.0.1:56054)
0810 07:29:52.398747 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:52.398763 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:52.399631 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:52.401385 (+ 1754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:52.401390 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:52.401571 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:52.401678 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:52.401687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.403478 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.403482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:52.404260 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:52.404272 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.404861 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.404865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:52.405015 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:52.405528 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:52.405542 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:52.407269 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:52.409800 (+ 2531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:52.409805 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:52.409807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:52.410104 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:52.410437 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:52.410440 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:52.410451 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:52.410542 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:52.410993 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:52.410996 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:52.411097 (+ 101us) client_negotiation.cc:769] Sending connection context
0810 07:29:52.411181 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 07:29:52.411369 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":1}
I20260810 07:29:52.412468 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.401231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:34291 (local address 127.0.0.1:56068)
0810 07:29:52.401366 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:52.401380 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:52.401443 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:52.402461 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:52.402464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:52.402965 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:52.403055 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:52.403065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.404086 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.404089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:52.404867 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:52.404879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.405809 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.405812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:52.405904 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:52.406924 (+ 1020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:52.406939 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:52.408433 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:52.410851 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:52.410856 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:52.410861 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:52.411123 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:52.411418 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:52.411422 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:52.411424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:52.411509 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:52.411938 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:52.411941 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:52.412028 (+ 87us) client_negotiation.cc:769] Sending connection context
0810 07:29:52.412139 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 07:29:52.412294 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 07:29:52.412518 16656 log.cc:824] T 00000000000000000000000000000000 P 307516bf5ae545dfaef76bfb6dba105e: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:52.412514 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.401742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56068 (local address 127.12.157.252:34291)
0810 07:29:52.402079 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:52.402091 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:52.402107 (+ 16us) server_negotiation.cc:407] Connection header received
0810 07:29:52.402143 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:52.402147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:52.402192 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:52.402289 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:52.403171 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.403976 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.405031 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.405365 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.405713 (+ 348us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:52.408578 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:52.408583 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:52.408589 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:52.408614 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:52.410725 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:52.411233 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:52.411236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:52.411238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:52.411335 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:52.411626 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:52.411630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:52.411633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:52.411846 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:52.411936 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:52.412186 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 07:29:52.412360 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":273,"thread_start_us":98,"threads_started":1}
I20260810 07:29:52.413087 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.399131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56054 (local address 127.12.157.252:34291)
0810 07:29:52.401147 (+ 2016us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:52.401162 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:52.401173 (+ 11us) server_negotiation.cc:407] Connection header received
0810 07:29:52.401202 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:52.401205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:52.401249 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:52.401794 (+ 545us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:52.401910 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.403337 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.404409 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.404733 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.405248 (+ 515us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:52.407417 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:52.407422 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:52.407424 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:52.407462 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:52.409658 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:52.410239 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:52.410242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:52.410244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:52.410330 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:52.410660 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:52.410663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:52.410665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:52.410879 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:52.412658 (+ 1779us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:52.412751 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 07:29:52.412874 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":135,"threads_started":1}
I20260810 07:29:52.415048 16796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.402438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45514 (local address 127.12.157.253:46105)
0810 07:29:52.402773 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:52.402777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:52.402954 (+ 177us) server_negotiation.cc:407] Connection header received
0810 07:29:52.403146 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:52.403152 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:52.403195 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:52.403249 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:52.404091 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.404844 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.405909 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.406247 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.406331 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:52.409578 (+ 3247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:52.409585 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:52.409622 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:52.409654 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:52.412114 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:52.413288 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:52.413291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:52.413293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:52.413383 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:52.413851 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:52.413858 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:52.413860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:52.414114 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:52.414303 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:52.414774 (+ 471us) server_negotiation.cc:299] Negotiation successful
0810 07:29:52.414903 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":254}
I20260810 07:29:52.415671 16656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 307516bf5ae545dfaef76bfb6dba105e: No bootstrap required, opened a new log
I20260810 07:29:52.416118 16785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 275c970eda6046e190aff105bcb41dbe: Bootstrap starting.
I20260810 07:29:52.416118 16915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c127907788fd41949e62f0734353f674: Bootstrap starting.
I20260810 07:29:52.418937 16785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 275c970eda6046e190aff105bcb41dbe: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:52.418937 16915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c127907788fd41949e62f0734353f674: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:52.419087 16656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 307516bf5ae545dfaef76bfb6dba105e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307516bf5ae545dfaef76bfb6dba105e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46277 } } peers { permanent_uuid: "275c970eda6046e190aff105bcb41dbe" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 46105 } } peers { permanent_uuid: "c127907788fd41949e62f0734353f674" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 34291 } }
I20260810 07:29:52.419272 16656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 307516bf5ae545dfaef76bfb6dba105e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 307516bf5ae545dfaef76bfb6dba105e, State: Initialized, Role: FOLLOWER
I20260810 07:29:52.419529 16656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 307516bf5ae545dfaef76bfb6dba105e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307516bf5ae545dfaef76bfb6dba105e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46277 } } peers { permanent_uuid: "275c970eda6046e190aff105bcb41dbe" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 46105 } } peers { permanent_uuid: "c127907788fd41949e62f0734353f674" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 34291 } }
I20260810 07:29:52.419852 16915 log.cc:824] T 00000000000000000000000000000000 P c127907788fd41949e62f0734353f674: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:52.419852 16785 log.cc:824] T 00000000000000000000000000000000 P 275c970eda6046e190aff105bcb41dbe: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:52.419988 16928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307516bf5ae545dfaef76bfb6dba105e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307516bf5ae545dfaef76bfb6dba105e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46277 } } peers { permanent_uuid: "275c970eda6046e190aff105bcb41dbe" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 46105 } } peers { permanent_uuid: "c127907788fd41949e62f0734353f674" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 34291 } } }
I20260810 07:29:52.420109 16928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307516bf5ae545dfaef76bfb6dba105e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:29:52.421137 16656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 307516bf5ae545dfaef76bfb6dba105e [sys.catalog]: configured and running, proceeding with master startup.
W20260810 07:29:52.422367 16571 tablet.cc:2369] T 00000000000000000000000000000000 P 307516bf5ae545dfaef76bfb6dba105e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 07:29:52.422608 16785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 275c970eda6046e190aff105bcb41dbe: No bootstrap required, opened a new log
I20260810 07:29:52.422719 16915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c127907788fd41949e62f0734353f674: No bootstrap required, opened a new log
I20260810 07:29:52.424105 12919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:29:52.425554 16607 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "307516bf5ae545dfaef76bfb6dba105e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50130
I20260810 07:29:52.425782 16607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 307516bf5ae545dfaef76bfb6dba105e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 07:29:52.425889 16607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 307516bf5ae545dfaef76bfb6dba105e [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:52.426934 16785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 275c970eda6046e190aff105bcb41dbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307516bf5ae545dfaef76bfb6dba105e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46277 } } peers { permanent_uuid: "275c970eda6046e190aff105bcb41dbe" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 46105 } } peers { permanent_uuid: "c127907788fd41949e62f0734353f674" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 34291 } }
I20260810 07:29:52.426941 16915 raft_consensus.cc:348] T 00000000000000000000000000000000 P c127907788fd41949e62f0734353f674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307516bf5ae545dfaef76bfb6dba105e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46277 } } peers { permanent_uuid: "275c970eda6046e190aff105bcb41dbe" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 46105 } } peers { permanent_uuid: "c127907788fd41949e62f0734353f674" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 34291 } }
I20260810 07:29:52.427153 16785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 275c970eda6046e190aff105bcb41dbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 275c970eda6046e190aff105bcb41dbe, State: Initialized, Role: FOLLOWER
I20260810 07:29:52.427188 16915 raft_consensus.cc:729] T 00000000000000000000000000000000 P c127907788fd41949e62f0734353f674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c127907788fd41949e62f0734353f674, State: Initialized, Role: FOLLOWER
I20260810 07:29:52.427284 16607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 307516bf5ae545dfaef76bfb6dba105e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307516bf5ae545dfaef76bfb6dba105e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46277 } } peers { permanent_uuid: "275c970eda6046e190aff105bcb41dbe" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 46105 } } peers { permanent_uuid: "c127907788fd41949e62f0734353f674" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 34291 } }
I20260810 07:29:52.427497 16915 consensus_queue.cc:260] T 00000000000000000000000000000000 P c127907788fd41949e62f0734353f674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307516bf5ae545dfaef76bfb6dba105e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46277 } } peers { permanent_uuid: "275c970eda6046e190aff105bcb41dbe" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 46105 } } peers { permanent_uuid: "c127907788fd41949e62f0734353f674" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 34291 } }
I20260810 07:29:52.427476 16785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 275c970eda6046e190aff105bcb41dbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307516bf5ae545dfaef76bfb6dba105e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46277 } } peers { permanent_uuid: "275c970eda6046e190aff105bcb41dbe" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 46105 } } peers { permanent_uuid: "c127907788fd41949e62f0734353f674" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 34291 } }
I20260810 07:29:52.428192 16607 leader_election.cc:288] T 00000000000000000000000000000000 P 307516bf5ae545dfaef76bfb6dba105e [CANDIDATE]: Term 1 election: Requested vote from peers 275c970eda6046e190aff105bcb41dbe (127.12.157.253:46105), c127907788fd41949e62f0734353f674 (127.12.157.252:34291)
I20260810 07:29:52.428901 16785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 275c970eda6046e190aff105bcb41dbe [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:52.429561 16736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "307516bf5ae545dfaef76bfb6dba105e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "275c970eda6046e190aff105bcb41dbe"
I20260810 07:29:52.429742 16915 sys_catalog.cc:564] T 00000000000000000000000000000000 P c127907788fd41949e62f0734353f674 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:52.429800 16736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 275c970eda6046e190aff105bcb41dbe [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:52.426927 16533 master_runner.cc:186] Error getting master registration for 127.12.157.253:46105: OK, instance_id { permanent_uuid: "275c970eda6046e190aff105bcb41dbe" instance_seqno: 1786346992137347 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 07:29:52.431137 16533 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 07:29:52.431663 16736 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 275c970eda6046e190aff105bcb41dbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 307516bf5ae545dfaef76bfb6dba105e in term 1.
I20260810 07:29:52.432183 16952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 275c970eda6046e190aff105bcb41dbe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307516bf5ae545dfaef76bfb6dba105e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46277 } } peers { permanent_uuid: "275c970eda6046e190aff105bcb41dbe" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 46105 } } peers { permanent_uuid: "c127907788fd41949e62f0734353f674" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 34291 } } }
I20260810 07:29:52.432559 16952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 275c970eda6046e190aff105bcb41dbe [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:29:52.432767 16871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "307516bf5ae545dfaef76bfb6dba105e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c127907788fd41949e62f0734353f674"
I20260810 07:29:52.432991 16563 leader_election.cc:302] T 00000000000000000000000000000000 P 307516bf5ae545dfaef76bfb6dba105e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 275c970eda6046e190aff105bcb41dbe, 307516bf5ae545dfaef76bfb6dba105e; no voters:
I20260810 07:29:52.433066 16871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c127907788fd41949e62f0734353f674 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:52.433329 16928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 307516bf5ae545dfaef76bfb6dba105e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:52.433723 16928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 307516bf5ae545dfaef76bfb6dba105e [term 1 LEADER]: Becoming Leader. State: Replica: 307516bf5ae545dfaef76bfb6dba105e, State: Running, Role: LEADER
I20260810 07:29:52.434106 16928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 307516bf5ae545dfaef76bfb6dba105e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307516bf5ae545dfaef76bfb6dba105e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46277 } } peers { permanent_uuid: "275c970eda6046e190aff105bcb41dbe" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 46105 } } peers { permanent_uuid: "c127907788fd41949e62f0734353f674" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 34291 } }
I20260810 07:29:52.434949 16871 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c127907788fd41949e62f0734353f674 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 307516bf5ae545dfaef76bfb6dba105e in term 1.
I20260810 07:29:52.435688 16947 sys_catalog.cc:455] T 00000000000000000000000000000000 P c127907788fd41949e62f0734353f674 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307516bf5ae545dfaef76bfb6dba105e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46277 } } peers { permanent_uuid: "275c970eda6046e190aff105bcb41dbe" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 46105 } } peers { permanent_uuid: "c127907788fd41949e62f0734353f674" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 34291 } } }
I20260810 07:29:52.435864 16947 sys_catalog.cc:458] T 00000000000000000000000000000000 P c127907788fd41949e62f0734353f674 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 07:29:52.438350 16945 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 307516bf5ae545dfaef76bfb6dba105e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:29:52.438421 16945 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260810 07:29:52.438426 16998 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 275c970eda6046e190aff105bcb41dbe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:29:52.438499 16998 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:29:52.439807 16984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307516bf5ae545dfaef76bfb6dba105e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 307516bf5ae545dfaef76bfb6dba105e. Latest consensus state: current_term: 1 leader_uuid: "307516bf5ae545dfaef76bfb6dba105e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307516bf5ae545dfaef76bfb6dba105e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46277 } } peers { permanent_uuid: "275c970eda6046e190aff105bcb41dbe" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 46105 } } peers { permanent_uuid: "c127907788fd41949e62f0734353f674" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 34291 } } }
I20260810 07:29:52.440047 16984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307516bf5ae545dfaef76bfb6dba105e [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:52.440495 17008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:52.441524 17008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260810 07:29:52.442477 16983 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c127907788fd41949e62f0734353f674: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:29:52.442556 16983 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:29:52.454476 16796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.438881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45520 (local address 127.12.157.253:46105)
0810 07:29:52.439008 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:52.439013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:52.444505 (+ 5492us) server_negotiation.cc:407] Connection header received
0810 07:29:52.444720 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:52.444724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:52.444780 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:52.444834 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:52.445998 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.446808 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.448014 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.448370 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.448446 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:52.450415 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:52.450419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:52.450422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:52.450450 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:52.452361 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:52.452914 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:52.452923 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:52.452926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:52.453044 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:52.453381 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:52.453385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:52.453387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:52.453611 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:52.453724 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:52.454108 (+ 384us) server_negotiation.cc:299] Negotiation successful
0810 07:29:52.454260 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 07:29:52.460774 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.439345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56078 (local address 127.12.157.252:34291)
0810 07:29:52.439459 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:52.439464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:52.445225 (+ 5761us) server_negotiation.cc:407] Connection header received
0810 07:29:52.445629 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:52.445635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:52.445692 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:52.445758 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:52.447864 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.448632 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.451970 (+ 3338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.452386 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.452468 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:52.456345 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:52.456350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:52.456352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:52.456384 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:52.458722 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:52.459242 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:52.459246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:52.459247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:52.459350 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:52.459670 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:52.459673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:52.459675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:52.459924 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:52.460061 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:52.460440 (+ 379us) server_negotiation.cc:299] Negotiation successful
0810 07:29:52.460598 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 07:29:52.461045 16661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:52.443658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50146 (local address 127.12.157.254:46277)
0810 07:29:52.443795 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:52.443800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:52.445542 (+ 1742us) server_negotiation.cc:407] Connection header received
0810 07:29:52.446799 (+ 1257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:52.446803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:52.446858 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:52.446909 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:52.449840 (+ 2931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.450575 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.453158 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:52.453504 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:52.453579 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:52.457404 (+ 3825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:52.457409 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:52.457413 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:52.457448 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:52.459125 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:52.459602 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:52.459608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:52.459609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:52.459701 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:52.460025 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:52.460028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:52.460030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:52.460243 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:52.460375 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:52.460709 (+ 334us) server_negotiation.cc:299] Negotiation successful
0810 07:29:52.460849 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 07:29:52.468688 16871 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c127907788fd41949e62f0734353f674 [term 1 FOLLOWER]: Refusing update from remote peer 307516bf5ae545dfaef76bfb6dba105e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:29:52.468782 16736 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 275c970eda6046e190aff105bcb41dbe [term 1 FOLLOWER]: Refusing update from remote peer 307516bf5ae545dfaef76bfb6dba105e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:29:52.469411 16984 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 307516bf5ae545dfaef76bfb6dba105e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c127907788fd41949e62f0734353f674" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 34291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:29:52.469852 16984 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 307516bf5ae545dfaef76bfb6dba105e [LEADER]: Connected to new peer: Peer: permanent_uuid: "275c970eda6046e190aff105bcb41dbe" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 46105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:29:52.473560 16947 sys_catalog.cc:455] T 00000000000000000000000000000000 P c127907788fd41949e62f0734353f674 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 307516bf5ae545dfaef76bfb6dba105e. Latest consensus state: current_term: 1 leader_uuid: "307516bf5ae545dfaef76bfb6dba105e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307516bf5ae545dfaef76bfb6dba105e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46277 } } peers { permanent_uuid: "275c970eda6046e190aff105bcb41dbe" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 46105 } } peers { permanent_uuid: "c127907788fd41949e62f0734353f674" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 34291 } } }
I20260810 07:29:52.473676 16947 sys_catalog.cc:458] T 00000000000000000000000000000000 P c127907788fd41949e62f0734353f674 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:29:52.473816 16952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 275c970eda6046e190aff105bcb41dbe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 307516bf5ae545dfaef76bfb6dba105e. Latest consensus state: current_term: 1 leader_uuid: "307516bf5ae545dfaef76bfb6dba105e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307516bf5ae545dfaef76bfb6dba105e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46277 } } peers { permanent_uuid: "275c970eda6046e190aff105bcb41dbe" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 46105 } } peers { permanent_uuid: "c127907788fd41949e62f0734353f674" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 34291 } } }
I20260810 07:29:52.473928 16952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 275c970eda6046e190aff105bcb41dbe [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:29:52.479574 16984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307516bf5ae545dfaef76bfb6dba105e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "307516bf5ae545dfaef76bfb6dba105e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307516bf5ae545dfaef76bfb6dba105e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46277 } } peers { permanent_uuid: "275c970eda6046e190aff105bcb41dbe" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 46105 } } peers { permanent_uuid: "c127907788fd41949e62f0734353f674" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 34291 } } }
I20260810 07:29:52.479698 16984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307516bf5ae545dfaef76bfb6dba105e [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:52.480696 17015 mvcc.cc:205] Tried to move back new op lower bound from 7316877281144307712 to 7316877281014632448. Current Snapshot: MvccSnapshot[applied={T|T < 7316877281144307712}]
I20260810 07:29:52.480896 16947 sys_catalog.cc:455] T 00000000000000000000000000000000 P c127907788fd41949e62f0734353f674 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "307516bf5ae545dfaef76bfb6dba105e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307516bf5ae545dfaef76bfb6dba105e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46277 } } peers { permanent_uuid: "275c970eda6046e190aff105bcb41dbe" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 46105 } } peers { permanent_uuid: "c127907788fd41949e62f0734353f674" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 34291 } } }
I20260810 07:29:52.481024 16947 sys_catalog.cc:458] T 00000000000000000000000000000000 P c127907788fd41949e62f0734353f674 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:29:52.481386 17009 mvcc.cc:205] Tried to move back new op lower bound from 7316877281144307712 to 7316877281014632448. Current Snapshot: MvccSnapshot[applied={T|T < 7316877281144307712}]
I20260810 07:29:52.483974 16928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307516bf5ae545dfaef76bfb6dba105e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "307516bf5ae545dfaef76bfb6dba105e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307516bf5ae545dfaef76bfb6dba105e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46277 } } peers { permanent_uuid: "275c970eda6046e190aff105bcb41dbe" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 46105 } } peers { permanent_uuid: "c127907788fd41949e62f0734353f674" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 34291 } } }
I20260810 07:29:52.484102 16928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307516bf5ae545dfaef76bfb6dba105e [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:52.485204 16952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 275c970eda6046e190aff105bcb41dbe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "307516bf5ae545dfaef76bfb6dba105e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307516bf5ae545dfaef76bfb6dba105e" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 46277 } } peers { permanent_uuid: "275c970eda6046e190aff105bcb41dbe" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 46105 } } peers { permanent_uuid: "c127907788fd41949e62f0734353f674" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 34291 } } }
I20260810 07:29:52.485327 16952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 275c970eda6046e190aff105bcb41dbe [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:29:52.486047 17017 mvcc.cc:205] Tried to move back new op lower bound from 7316877281144307712 to 7316877281014632448. Current Snapshot: MvccSnapshot[applied={T|T < 7316877281144307712}]
I20260810 07:29:52.490486 17008 catalog_manager.cc:1347] Generated new cluster ID: 6e35ab79b6b747719f9a88a1809774e6
I20260810 07:29:52.490566 17008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:52.509063 17008 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:52.510417 17008 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:52.529385 17008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 307516bf5ae545dfaef76bfb6dba105e: Generated new TSK 0
I20260810 07:29:52.530370 17008 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 07:29:52.569412 16661 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:52.566827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50156 (local address 127.12.157.254:46277)
0810 07:29:52.566964 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:52.566969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:52.568231 (+ 1262us) server_negotiation.cc:407] Connection header received
0810 07:29:52.568445 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:52.568449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:52.568506 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:52.568566 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:52.569262 (+ 696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50156: BlockingRecv error: recv got EOF from 127.0.0.1:50156 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 07:29:52.570787 16923 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:52.567708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56080 (local address 127.12.157.252:34291)
0810 07:29:52.567859 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:52.567865 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:52.569899 (+ 2034us) server_negotiation.cc:407] Connection header received
0810 07:29:52.570069 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:52.570073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:52.570122 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:52.570172 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:52.570665 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56080: BlockingRecv error: recv got EOF from 127.0.0.1:56080 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 07:29:52.572379 16796 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:52.571442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45532 (local address 127.12.157.253:46105)
0810 07:29:52.571573 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:52.571578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:52.571641 (+ 63us) server_negotiation.cc:407] Connection header received
0810 07:29:52.571761 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:52.571764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:52.571811 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:52.571897 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:52.572245 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45532: BlockingRecv error: recv got EOF from 127.0.0.1:45532 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 07:29:52.576073 16796 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:52.574816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45546 (local address 127.12.157.253:46105)
0810 07:29:52.574926 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:52.574930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:52.575146 (+ 216us) server_negotiation.cc:407] Connection header received
0810 07:29:52.575361 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:52.575365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:52.575452 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:52.575505 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:52.575985 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45546: BlockingRecv error: recv got EOF from 127.0.0.1:45546 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260810 07:29:52.581038 16796 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:52.579553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45552 (local address 127.12.157.253:46105)
0810 07:29:52.579652 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:52.579656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:52.580142 (+ 486us) server_negotiation.cc:407] Connection header received
0810 07:29:52.580327 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:52.580331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:52.580378 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:52.580425 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:52.580916 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45552: BlockingRecv error: recv got EOF from 127.0.0.1:45552 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260810 07:29:52.586719 16796 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:52.585249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45566 (local address 127.12.157.253:46105)
0810 07:29:52.585345 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:52.585349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:52.585810 (+ 461us) server_negotiation.cc:407] Connection header received
0810 07:29:52.585990 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:52.585993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:52.586041 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:52.586091 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:52.586610 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45566: BlockingRecv error: recv got EOF from 127.0.0.1:45566 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260810 07:29:52.594666 16796 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:52.593283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45574 (local address 127.12.157.253:46105)
0810 07:29:52.593433 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:52.593438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:52.593720 (+ 282us) server_negotiation.cc:407] Connection header received
0810 07:29:52.593919 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:52.593923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:52.593973 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:52.594024 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:52.594558 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45574: BlockingRecv error: recv got EOF from 127.0.0.1:45574 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260810 07:29:52.606752 16796 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:52.605227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45588 (local address 127.12.157.253:46105)
0810 07:29:52.605373 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:52.605378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:52.605698 (+ 320us) server_negotiation.cc:407] Connection header received
0810 07:29:52.605909 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:52.605914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:52.605970 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:52.606025 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:52.606627 (+ 602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45588: BlockingRecv error: recv got EOF from 127.0.0.1:45588 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260810 07:29:52.627238 16796 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:52.625595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45592 (local address 127.12.157.253:46105)
0810 07:29:52.625717 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:52.625722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:52.626245 (+ 523us) server_negotiation.cc:407] Connection header received
0810 07:29:52.626467 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:52.626471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:52.626525 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:52.626576 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:52.627119 (+ 543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45592: BlockingRecv error: recv got EOF from 127.0.0.1:45592 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 07:29:52.663491 16796 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:52.662146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45594 (local address 127.12.157.253:46105)
0810 07:29:52.662281 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:52.662286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:52.662559 (+ 273us) server_negotiation.cc:407] Connection header received
0810 07:29:52.662719 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:52.662723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:52.662779 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:52.662877 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:52.663362 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45594: BlockingRecv error: recv got EOF from 127.0.0.1:45594 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 07:29:52.731676 16796 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:52.730146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45606 (local address 127.12.157.253:46105)
0810 07:29:52.730322 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:52.730327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:52.730625 (+ 298us) server_negotiation.cc:407] Connection header received
0810 07:29:52.730853 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:52.730856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:52.730910 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:52.730988 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:52.731546 (+ 558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45606: BlockingRecv error: recv got EOF from 127.0.0.1:45606 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260810 07:29:52.863204 16796 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:52.861959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45614 (local address 127.12.157.253:46105)
0810 07:29:52.862149 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:52.862152 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:52.862240 (+ 88us) server_negotiation.cc:407] Connection header received
0810 07:29:52.862367 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:52.862370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:52.862420 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:52.862496 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:52.863052 (+ 556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45614: BlockingRecv error: recv got EOF from 127.0.0.1:45614 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260810 07:29:53.122788 16796 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:53.121545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45630 (local address 127.12.157.253:46105)
0810 07:29:53.121702 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:53.121707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:53.121831 (+ 124us) server_negotiation.cc:407] Connection header received
0810 07:29:53.122009 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:53.122013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:53.122070 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:53.122160 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:53.122664 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45630: BlockingRecv error: recv got EOF from 127.0.0.1:45630 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 07:29:53.439870 16998 catalog_manager.cc:1259] Loaded cluster ID: 6e35ab79b6b747719f9a88a1809774e6
I20260810 07:29:53.439956 16998 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 275c970eda6046e190aff105bcb41dbe: loading cluster ID for follower catalog manager: success
I20260810 07:29:53.441709 16998 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 275c970eda6046e190aff105bcb41dbe: acquiring CA information for follower catalog manager: success
I20260810 07:29:53.442814 16998 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 275c970eda6046e190aff105bcb41dbe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 07:29:53.443682 16983 catalog_manager.cc:1259] Loaded cluster ID: 6e35ab79b6b747719f9a88a1809774e6
I20260810 07:29:53.443768 16983 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c127907788fd41949e62f0734353f674: loading cluster ID for follower catalog manager: success
I20260810 07:29:53.445426 16983 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c127907788fd41949e62f0734353f674: acquiring CA information for follower catalog manager: success
I20260810 07:29:53.446532 16983 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c127907788fd41949e62f0734353f674: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 07:29:53.641153 17125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:53.637044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45644 (local address 127.12.157.253:46105)
0810 07:29:53.637305 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:53.637310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:53.637399 (+ 89us) server_negotiation.cc:407] Connection header received
0810 07:29:53.637577 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:53.637581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:53.637724 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:53.637881 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:29:53.638146 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:53.638678 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:53.639574 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:53.639826 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:53.639932 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:53.640256 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:29:53.640718 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:29:53.640867 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:53.640902 (+ 35us) server_negotiation.cc:299] Negotiation successful
0810 07:29:53.640991 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":83,"threads_started":1}
I20260810 07:29:53.647748 17133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:53.644150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56092 (local address 127.12.157.252:34291)
0810 07:29:53.644374 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:53.644377 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:53.644390 (+ 13us) server_negotiation.cc:407] Connection header received
0810 07:29:53.644472 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:53.644474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:53.644582 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:53.644678 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:29:53.644866 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:53.645428 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:53.646319 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:53.646574 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:53.646667 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:53.646963 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:29:53.647385 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:29:53.647479 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:53.647574 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 07:29:53.647624 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":69,"threads_started":1}
I20260810 07:29:53.654819 17125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:53.650802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45658 (local address 127.12.157.253:46105)
0810 07:29:53.650948 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:53.650952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:53.651341 (+ 389us) server_negotiation.cc:407] Connection header received
0810 07:29:53.651520 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:53.651523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:53.651571 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:53.651644 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:29:53.652025 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:53.652798 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:53.653740 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:53.653992 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:53.654099 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:53.654352 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:29:53.654440 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:29:53.654524 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:53.654621 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 07:29:53.654676 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 07:29:53.655267 17133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:53.650850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56094 (local address 127.12.157.252:34291)
0810 07:29:53.650967 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:53.650972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:53.651263 (+ 291us) server_negotiation.cc:407] Connection header received
0810 07:29:53.651434 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:53.651437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:53.651508 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:53.651589 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:29:53.651878 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:53.652661 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:53.653997 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:53.654311 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:53.654418 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:53.654666 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:29:53.654741 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:29:53.654793 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:53.655092 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 07:29:53.655146 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 07:29:53.655946 17147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:53.650933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50170 (local address 127.12.157.254:46277)
0810 07:29:53.651194 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:53.651200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:53.651217 (+ 17us) server_negotiation.cc:407] Connection header received
0810 07:29:53.651446 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:53.651450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:53.651606 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:53.651719 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:29:53.651970 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:53.652510 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:53.654029 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:53.654396 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:53.654502 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:53.654914 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:29:53.655259 (+ 345us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:29:53.655358 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:53.655755 (+ 397us) server_negotiation.cc:299] Negotiation successful
0810 07:29:53.655819 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":86,"threads_started":1}
I20260810 07:29:53.659644 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 16533
I20260810 07:29:53.674340 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 16668
I20260810 07:29:53.688073 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 16800
2026-08-10T07:29:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3365 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 07:29:53 dist-test-slave-ffrd krb5kdc[17152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 07:29:53 dist-test-slave-ffrd krb5kdc[17152](info): set up 2 sockets
Aug 10 07:29:53 dist-test-slave-ffrd krb5kdc[17152](info): commencing operation
krb5kdc: starting...
W20260810 07:29:54.974058 12919 mini_kdc.cc:121] Time spent starting KDC: real 1.253s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/krb5kdc/test-user.keytab.
Aug 10 07:29:55 dist-test-slave-ffrd krb5kdc[17152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T07:29:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T07:29:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.157.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
Entry for principal kudu/127.12.157.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab.
I20260810 07:29:55.103860 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:37121
--webserver_interface=127.12.157.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:37121,127.12.157.253:33729,127.12.157.252:42589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 07:29:55.220324 17181 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 07:29:55.220657 17181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:55.220724 17181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:55.220814 17181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:55.224812 17181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:55.224900 17181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:55.224959 17181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:55.224997 17181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:55.225031 17181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:55.230259 17181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:37121,127.12.157.253:33729,127.12.157.252:42589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.254.keytab
--principal=kudu/127.12.157.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.254:37121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.157.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:55.231345 17181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:55.232354 17181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:55.240796 17196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:55.240801 17189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:55.240801 17187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:55.241178 17181 server_base.cc:1034] running on GCE node
I20260810 07:29:55.241745 17181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:55.242748 17181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:55.243932 17181 hybrid_clock.cc:648] HybridClock initialized: now 1786346995243896 us; error 29 us; skew 500 ppm
Aug 10 07:29:55 dist-test-slave-ffrd krb5kdc[17152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346995, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:55.249886 17181 init.cc:377] Logged in from keytab as kudu/127.12.157.254@KRBTEST.COM (short username kudu)
I20260810 07:29:55.250805 17181 webserver.cc:460] Webserver started at http://127.12.157.254:41505/ using document root <none> and password file <none>
I20260810 07:29:55.251360 17181 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:55.251423 17181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:55.251638 17181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:55.253748 17181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-0/data/instance:
uuid: "d39d9726d60641579229d0fb307e4239"
format_stamp: "Formatted at 2026-08-10 07:29:55 on dist-test-slave-ffrd"
server_key: "1aa5b66fd238311249d483ee11a0e434"
server_key_iv: "1f6b58645b1c8448301f07bb1784329d"
server_key_version: "encryptionkey@0"
I20260810 07:29:55.254309 17181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-0/wal/instance:
uuid: "d39d9726d60641579229d0fb307e4239"
format_stamp: "Formatted at 2026-08-10 07:29:55 on dist-test-slave-ffrd"
server_key: "1aa5b66fd238311249d483ee11a0e434"
server_key_iv: "1f6b58645b1c8448301f07bb1784329d"
server_key_version: "encryptionkey@0"
I20260810 07:29:55.257963 17181 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 07:29:55.260378 17207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:55.261641 17181 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 07:29:55.261781 17181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-0/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-0/wal
uuid: "d39d9726d60641579229d0fb307e4239"
format_stamp: "Formatted at 2026-08-10 07:29:55 on dist-test-slave-ffrd"
server_key: "1aa5b66fd238311249d483ee11a0e434"
server_key_iv: "1f6b58645b1c8448301f07bb1784329d"
server_key_version: "encryptionkey@0"
I20260810 07:29:55.261895 17181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:55.276428 17181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:55.279342 17181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:55.279558 17181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:55.286634 17181 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.254:37121
I20260810 07:29:55.286933 17287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.254:37121 every 8 connection(s)
I20260810 07:29:55.287635 17181 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-0/data/info.pb
I20260810 07:29:55.290391 17288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:55.291213 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 17181
I20260810 07:29:55.291347 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-0/wal/instance
I20260810 07:29:55.291654 12919 external_mini_cluster.cc:1428] Setting key 308f9c45f8121b3863fea9c43b8ace1e
I20260810 07:29:55.295367 17288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37121 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.157.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.253@KRBTEST.COM" created.
Aug 10 07:29:55 dist-test-slave-ffrd krb5kdc[17152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346995, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:29:55.311028 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.297771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58190 (local address 127.12.157.254:37121)
0810 07:29:55.298297 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:55.298315 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:55.298546 (+ 231us) server_negotiation.cc:407] Connection header received
0810 07:29:55.299553 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:55.299561 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:55.299822 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:55.300051 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:55.300798 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.301821 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.302888 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.303352 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.303516 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:55.306004 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:55.306015 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:55.306033 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:55.306062 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:55.307749 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.308373 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.308378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.308380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.308510 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.308895 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.308897 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.308898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.309329 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:55.309543 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:55.310006 (+ 463us) server_negotiation.cc:299] Negotiation successful
0810 07:29:55.310237 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":338,"thread_start_us":142,"threads_started":1}
I20260810 07:29:55.311033 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.297768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37121 (local address 127.0.0.1:58190)
0810 07:29:55.298375 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:55.298421 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:55.299231 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:55.300043 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:55.300047 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:55.300353 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:55.300646 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:55.300659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.301940 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.301943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:55.302742 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:55.302781 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.303498 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.303502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:55.303655 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:55.304255 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:55.304272 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:55.305849 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:55.307867 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:55.307881 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:55.307893 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:55.308222 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:55.308620 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:55.308626 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:55.308630 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:55.308795 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:55.309516 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:55.309525 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:55.309706 (+ 181us) client_negotiation.cc:769] Sending connection context
0810 07:29:55.309917 (+ 211us) client_negotiation.cc:240] Negotiation successful
0810 07:29:55.310347 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":125,"spinlock_wait_cycles":52992,"thread_start_us":160,"threads_started":1}
I20260810 07:29:55.313890 17288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33729 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:29:55.314736 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.314456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:33729 (local address 127.0.0.1:55564)
0810 07:29:55.314598 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:55.314672 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.253:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 07:29:55.315060 17213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.157.253:33729: connect: Connection refused (error 111)
I20260810 07:29:55.316031 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.315785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:33729 (local address 127.0.0.1:55580)
0810 07:29:55.315885 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:55.315939 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.253:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 07:29:55.316218 17288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.157.253:33729: Network error: Client connection negotiation failed: client connection to 127.12.157.253:33729: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.253.keytab.
Entry for principal kudu/127.12.157.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.253.keytab.
I20260810 07:29:55.320317 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.253:33729
--webserver_interface=127.12.157.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.253.keytab
--principal=kudu/127.12.157.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:37121,127.12.157.253:33729,127.12.157.252:42589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:29:55.365432 17288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33729 } attempt: 1
I20260810 07:29:55.366360 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.366058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:33729 (local address 127.0.0.1:55582)
0810 07:29:55.366199 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:55.366268 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.253:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 07:29:55.367117 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.366905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:33729 (local address 127.0.0.1:55588)
0810 07:29:55.367013 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:55.367060 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.253:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 07:29:55.367266 17288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.157.253:33729: Network error: Client connection negotiation failed: client connection to 127.12.157.253:33729: connect: Connection refused (error 111)
W20260810 07:29:55.435189 17300 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 07:29:55.435487 17300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:55.435511 17300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:55.435588 17300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 07:29:55.435470 17288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33729 } attempt: 2
I20260810 07:29:55.436337 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.436039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:33729 (local address 127.0.0.1:55592)
0810 07:29:55.436187 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:55.436255 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.253:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 07:29:55.437242 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.436974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:33729 (local address 127.0.0.1:55596)
0810 07:29:55.437119 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:55.437177 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.253:33729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 07:29:55.437407 17288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.157.253:33729: Network error: Client connection negotiation failed: client connection to 127.12.157.253:33729: connect: Connection refused (error 111)
W20260810 07:29:55.440047 17300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:55.440102 17300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:55.440124 17300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:55.440140 17300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:55.440193 17300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:55.445721 17300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:37121,127.12.157.253:33729,127.12.157.252:42589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.253.keytab
--principal=kudu/127.12.157.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.253:33729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.157.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:55.446921 17300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:55.448026 17300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 07:29:55.456960 17310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:55.457006 17309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:55.457216 17300 server_base.cc:1034] running on GCE node
W20260810 07:29:55.457437 17312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:55.458053 17300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:55.459182 17300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:55.460381 17300 hybrid_clock.cc:648] HybridClock initialized: now 1786346995460360 us; error 33 us; skew 500 ppm
Aug 10 07:29:55 dist-test-slave-ffrd krb5kdc[17152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346995, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:55.463541 17300 init.cc:377] Logged in from keytab as kudu/127.12.157.253@KRBTEST.COM (short username kudu)
I20260810 07:29:55.464494 17300 webserver.cc:460] Webserver started at http://127.12.157.253:40693/ using document root <none> and password file <none>
I20260810 07:29:55.465191 17300 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:55.465284 17300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:55.465562 17300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:55.467901 17300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-1/data/instance:
uuid: "ca53018f8e8b4b51ae3a1e86dcce5286"
format_stamp: "Formatted at 2026-08-10 07:29:55 on dist-test-slave-ffrd"
server_key: "a7e7907ea82a0d9057e89b64b6cbde42"
server_key_iv: "33051be71627a660f0630277924acd1a"
server_key_version: "encryptionkey@0"
I20260810 07:29:55.468497 17300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-1/wal/instance:
uuid: "ca53018f8e8b4b51ae3a1e86dcce5286"
format_stamp: "Formatted at 2026-08-10 07:29:55 on dist-test-slave-ffrd"
server_key: "a7e7907ea82a0d9057e89b64b6cbde42"
server_key_iv: "33051be71627a660f0630277924acd1a"
server_key_version: "encryptionkey@0"
I20260810 07:29:55.472446 17300 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 07:29:55.475111 17323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:55.476395 17300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 07:29:55.476555 17300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-1/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-1/wal
uuid: "ca53018f8e8b4b51ae3a1e86dcce5286"
format_stamp: "Formatted at 2026-08-10 07:29:55 on dist-test-slave-ffrd"
server_key: "a7e7907ea82a0d9057e89b64b6cbde42"
server_key_iv: "33051be71627a660f0630277924acd1a"
server_key_version: "encryptionkey@0"
I20260810 07:29:55.476670 17300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:55.487105 17300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:55.491532 17300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:55.491782 17300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:55.499344 17300 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.253:33729
I20260810 07:29:55.499780 17409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.253:33729 every 8 connection(s)
I20260810 07:29:55.500633 17300 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-1/data/info.pb
I20260810 07:29:55.503731 17412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:55.507694 17412 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37121 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:29:55.508330 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 17300
I20260810 07:29:55.508481 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-1/wal/instance
I20260810 07:29:55.508742 12919 external_mini_cluster.cc:1428] Setting key 8dcdba54820027ba7dc2b14e9ce1f468
Aug 10 07:29:55 dist-test-slave-ffrd krb5kdc[17152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346995, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.253@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.157.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.157.252@KRBTEST.COM" created.
I20260810 07:29:55.523758 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.511016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58204 (local address 127.12.157.254:37121)
0810 07:29:55.511192 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:55.511199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:55.511221 (+ 22us) server_negotiation.cc:407] Connection header received
0810 07:29:55.511930 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:55.511934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:55.512005 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:55.512113 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:55.513330 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.514198 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.515475 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.515744 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.515832 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:55.518896 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:55.518899 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:55.518901 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:55.518927 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:55.520873 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.521540 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.521545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.521548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.521673 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.522551 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.522555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.522557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.522754 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:55.522869 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:55.523447 (+ 578us) server_negotiation.cc:299] Negotiation successful
0810 07:29:55.523581 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 07:29:55.524650 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.510178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37121 (local address 127.0.0.1:58204)
0810 07:29:55.510746 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:55.510816 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:55.511710 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:55.512255 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:55.512262 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:55.512621 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:55.513179 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:55.513195 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.514347 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.514351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:55.515338 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:55.515348 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.515914 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.515920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:55.516178 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:55.516898 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:55.516925 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:55.518704 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:55.521015 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:55.521023 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:55.521040 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:55.521436 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:55.522275 (+ 839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:55.522278 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:55.522280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:55.522432 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:55.522867 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:55.522874 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:55.523168 (+ 294us) client_negotiation.cc:769] Sending connection context
0810 07:29:55.523367 (+ 199us) client_negotiation.cc:240] Negotiation successful
0810 07:29:55.523725 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":121,"threads_started":1}
I20260810 07:29:55.526129 17412 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33729 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:29:55.528586 17288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33729 } attempt: 3
Aug 10 07:29:55 dist-test-slave-ffrd krb5kdc[17152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346995, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.253@KRBTEST.COM for kudu/127.12.157.253@KRBTEST.COM
Aug 10 07:29:55 dist-test-slave-ffrd krb5kdc[17152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346995, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for kudu/127.12.157.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.157.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.252.keytab.
Entry for principal kudu/127.12.157.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.252.keytab.
I20260810 07:29:55.539803 12919 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
/tmp/dist-test-tasko6sLg5/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.252:42589
--webserver_interface=127.12.157.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.252.keytab
--principal=kudu/127.12.157.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.157.212:44805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.157.254:37121,127.12.157.253:33729,127.12.157.252:42589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 07:29:55.540419 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.529058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:33729 (local address 127.0.0.1:55608)
0810 07:29:55.529251 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:55.529268 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:55.529346 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:55.529610 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:55.529614 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:55.529813 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:55.529966 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:55.529974 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.530918 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.530922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:55.532035 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:55.532047 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.532528 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.532531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:55.532639 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:55.533177 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:55.533192 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:55.534946 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:55.538464 (+ 3518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:55.538467 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:55.538469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:55.538762 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:55.539143 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:55.539148 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:55.539151 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:55.539259 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:55.539853 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:55.539856 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:55.539970 (+ 114us) client_negotiation.cc:769] Sending connection context
0810 07:29:55.540077 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 07:29:55.540237 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260810 07:29:55.540575 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.529078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55608 (local address 127.12.157.253:33729)
0810 07:29:55.529390 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:55.529399 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:55.529411 (+ 12us) server_negotiation.cc:407] Connection header received
0810 07:29:55.529443 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:55.529446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:55.529484 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:55.529569 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:55.530080 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.530786 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.532184 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.532417 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.532523 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:55.535135 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:55.535138 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:55.535141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:55.535183 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:55.538334 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.538886 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.538892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.538893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.539011 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.539363 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.539367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.539369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.539734 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:55.539883 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:55.540207 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 07:29:55.540380 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":252,"spinlock_wait_cycles":4480,"thread_start_us":90,"threads_started":1}
I20260810 07:29:55.540717 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.526649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:33729 (local address 127.0.0.1:55598)
0810 07:29:55.526767 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:55.526782 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:55.526857 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:55.527530 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:55.527533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:55.527713 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:55.527795 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:55.527801 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.529024 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.529028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:55.529806 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:55.529814 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.530590 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.530596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:55.530726 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:55.531345 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:55.531364 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:55.534401 (+ 3037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:55.538907 (+ 4506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:55.538913 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:55.538918 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:55.539192 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:55.539517 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:55.539520 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:55.539522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:55.539602 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:55.540104 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:55.540109 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:55.540213 (+ 104us) client_negotiation.cc:769] Sending connection context
0810 07:29:55.540335 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 07:29:55.540501 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 07:29:55.541066 17419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.526802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55598 (local address 127.12.157.253:33729)
0810 07:29:55.527200 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:55.527209 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:55.527226 (+ 17us) server_negotiation.cc:407] Connection header received
0810 07:29:55.527257 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:55.527261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:55.527415 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:55.527551 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:55.527900 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.528713 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.529908 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.530448 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.530552 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:55.534557 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:55.534568 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:55.534583 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:55.534630 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:55.538310 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.539310 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.539313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.539315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.539414 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.539707 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.539711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.539714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.539959 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:55.540066 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:55.540658 (+ 592us) server_negotiation.cc:299] Negotiation successful
0810 07:29:55.540781 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":295,"thread_start_us":181,"threads_started":1}
I20260810 07:29:55.542948 17288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.252" port: 42589 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:29:55.543129 17412 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.252" port: 42589 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:29:55.543697 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.543453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:42589 (local address 127.0.0.1:34092)
0810 07:29:55.543560 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:55.543619 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 07:29:55.544260 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.543577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:42589 (local address 127.0.0.1:34104)
0810 07:29:55.543682 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:55.543899 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 07:29:55.544364 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.544093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:42589 (local address 127.0.0.1:34110)
0810 07:29:55.544237 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:55.544291 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260810 07:29:55.544482 17288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.157.252:42589: Network error: Client connection negotiation failed: client connection to 127.12.157.252:42589: connect: Connection refused (error 111)
W20260810 07:29:55.544452 17329 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.157.252:42589: connect: Connection refused (error 111)
I20260810 07:29:55.545410 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.545201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:42589 (local address 127.0.0.1:34126)
0810 07:29:55.545305 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:55.545358 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 07:29:55.545575 17412 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.157.252:42589: Network error: Client connection negotiation failed: client connection to 127.12.157.252:42589: connect: Connection refused (error 111)
I20260810 07:29:55.575632 17288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.157.252" port: 42589 } attempt: 1
I20260810 07:29:55.576609 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.576238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:42589 (local address 127.0.0.1:34142)
0810 07:29:55.576450 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:55.576523 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 07:29:55.577399 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.577167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:42589 (local address 127.0.0.1:34154)
0810 07:29:55.577272 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:55.577326 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 07:29:55.577521 17288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.157.252:42589: Network error: Client connection negotiation failed: client connection to 127.12.157.252:42589: connect: Connection refused (error 111)
I20260810 07:29:55.594825 17412 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.157.252" port: 42589 } attempt: 1
I20260810 07:29:55.595759 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.595409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:42589 (local address 127.0.0.1:34168)
0810 07:29:55.595588 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:55.595666 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 07:29:55.596635 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.596401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:42589 (local address 127.0.0.1:34178)
0810 07:29:55.596516 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:55.596570 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 07:29:55.596833 17412 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.157.252:42589: Network error: Client connection negotiation failed: client connection to 127.12.157.252:42589: connect: Connection refused (error 111)
I20260810 07:29:55.652714 17288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.157.252" port: 42589 } attempt: 2
I20260810 07:29:55.653667 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.653335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:42589 (local address 127.0.0.1:34188)
0810 07:29:55.653492 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:55.653569 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 07:29:55.654492 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.654199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:42589 (local address 127.0.0.1:34192)
0810 07:29:55.654371 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:55.654431 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260810 07:29:55.654639 17288 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.157.252:42589: Network error: Client connection negotiation failed: client connection to 127.12.157.252:42589: connect: Connection refused (error 111)
W20260810 07:29:55.659005 17421 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 07:29:55.659291 17421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 07:29:55.659363 17421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 07:29:55.659407 17421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 07:29:55.663627 17421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 07:29:55.663695 17421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 07:29:55.663715 17421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 07:29:55.663770 17421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 07:29:55.663806 17421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 07:29:55.665000 17412 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.157.252" port: 42589 } attempt: 2
I20260810 07:29:55.665856 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.665535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:42589 (local address 127.0.0.1:34202)
0810 07:29:55.665694 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:55.665763 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 07:29:55.666673 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.666361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:42589 (local address 127.0.0.1:34216)
0810 07:29:55.666568 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:55.666618 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.157.252:42589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260810 07:29:55.666821 17412 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.157.252:42589: Network error: Client connection negotiation failed: client connection to 127.12.157.252:42589: connect: Connection refused (error 111)
I20260810 07:29:55.670110 17421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.157.212:44805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.157.254:37121,127.12.157.253:33729,127.12.157.252:42589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/krb5kdc/kudu_127.12.157.252.keytab
--principal=kudu/127.12.157.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.157.252:42589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.157.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 06:33:02 UTC on ccycloud.quasar-daqefu.root.comops.site
build id 4577113
I20260810 07:29:55.671530 17421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 07:29:55.672783 17421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 07:29:55.681785 17421 server_base.cc:1034] running on GCE node
W20260810 07:29:55.681803 17432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:55.681815 17436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 07:29:55.681782 17430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 07:29:55.682621 17421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 07:29:55.683756 17421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 07:29:55.685003 17421 hybrid_clock.cc:648] HybridClock initialized: now 1786346995684964 us; error 46 us; skew 500 ppm
Aug 10 07:29:55 dist-test-slave-ffrd krb5kdc[17152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346995, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 07:29:55.687844 17421 init.cc:377] Logged in from keytab as kudu/127.12.157.252@KRBTEST.COM (short username kudu)
I20260810 07:29:55.688771 17421 webserver.cc:460] Webserver started at http://127.12.157.252:39975/ using document root <none> and password file <none>
I20260810 07:29:55.689399 17421 fs_manager.cc:359] Metadata directory not provided
I20260810 07:29:55.689491 17421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 07:29:55.689723 17421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 07:29:55.691818 17421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-2/data/instance:
uuid: "2f606e1479be49d885704ca1d4ffabdd"
format_stamp: "Formatted at 2026-08-10 07:29:55 on dist-test-slave-ffrd"
server_key: "64ca6cbfa3785a684d8a7efdc052422f"
server_key_iv: "6931f86b9b3978cb302d0829aee28a5f"
server_key_version: "encryptionkey@0"
I20260810 07:29:55.692406 17421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-2/wal/instance:
uuid: "2f606e1479be49d885704ca1d4ffabdd"
format_stamp: "Formatted at 2026-08-10 07:29:55 on dist-test-slave-ffrd"
server_key: "64ca6cbfa3785a684d8a7efdc052422f"
server_key_iv: "6931f86b9b3978cb302d0829aee28a5f"
server_key_version: "encryptionkey@0"
I20260810 07:29:55.696239 17421 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 07:29:55.699002 17444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 07:29:55.700227 17421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 07:29:55.700367 17421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-2/data,/tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-2/wal
uuid: "2f606e1479be49d885704ca1d4ffabdd"
format_stamp: "Formatted at 2026-08-10 07:29:55 on dist-test-slave-ffrd"
server_key: "64ca6cbfa3785a684d8a7efdc052422f"
server_key_iv: "6931f86b9b3978cb302d0829aee28a5f"
server_key_version: "encryptionkey@0"
I20260810 07:29:55.700485 17421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 07:29:55.727034 17421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 07:29:55.730322 17421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 07:29:55.730549 17421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 07:29:55.738160 17421 rpc_server.cc:307] RPC server started. Bound to: 127.12.157.252:42589
I20260810 07:29:55.738488 17521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.157.252:42589 every 8 connection(s)
I20260810 07:29:55.739233 17421 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-2/data/info.pb
I20260810 07:29:55.742548 17523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 07:29:55.746503 17523 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37121 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:29:55.749122 12919 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu as pid 17421
I20260810 07:29:55.749286 12919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasko6sLg5/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786346694975122-12919-0/minicluster-data/master-2/wal/instance
I20260810 07:29:55.749607 12919 external_mini_cluster.cc:1428] Setting key 4ee046958952704267a054d7ea786805
I20260810 07:29:55.753787 17288 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.157.252" port: 42589 } attempt: 3
Aug 10 07:29:55 dist-test-slave-ffrd krb5kdc[17152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346995, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.252@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
Aug 10 07:29:55 dist-test-slave-ffrd krb5kdc[17152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.252@KRBTEST.COM
I20260810 07:29:55.757995 17412 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.157.252" port: 42589 } attempt: 3
Aug 10 07:29:55 dist-test-slave-ffrd krb5kdc[17152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346995, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.254@KRBTEST.COM for kudu/127.12.157.252@KRBTEST.COM
I20260810 07:29:55.762481 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.748529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58220 (local address 127.12.157.254:37121)
0810 07:29:55.748695 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:55.748699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:55.749741 (+ 1042us) server_negotiation.cc:407] Connection header received
0810 07:29:55.750984 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:55.750988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:55.751045 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:55.751131 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:55.752127 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.752641 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.753716 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.754026 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.754132 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:55.757335 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:55.757341 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:55.757344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:55.757371 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:55.759400 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.760210 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.760213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.760215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.760304 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.761044 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.761050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.761054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.761392 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:55.761506 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:55.762168 (+ 662us) server_negotiation.cc:299] Negotiation successful
0810 07:29:55.762301 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 07:29:55.764161 17525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.748886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.254:37121 (local address 127.0.0.1:58220)
0810 07:29:55.749568 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:55.749615 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:55.750769 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:55.751206 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:55.751213 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:55.751582 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:55.751979 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:55.751993 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.752765 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.752770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:55.753587 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:55.753597 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.754181 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.754186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:55.754432 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:55.755136 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:55.755159 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:55.757137 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:55.759515 (+ 2378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:55.759525 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:55.759539 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:55.760092 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:55.760667 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:55.760669 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:55.760671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:55.760886 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:55.761582 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:55.761589 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:55.761866 (+ 277us) client_negotiation.cc:769] Sending connection context
0810 07:29:55.762648 (+ 782us) client_negotiation.cc:240] Negotiation successful
0810 07:29:55.763031 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":1,"thread_start_us":190,"threads_started":1}
Aug 10 07:29:55 dist-test-slave-ffrd krb5kdc[17152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346995, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.253@KRBTEST.COM for kudu/127.12.157.252@KRBTEST.COM
I20260810 07:29:55.764168 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.750314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34226 (local address 127.12.157.252:42589)
0810 07:29:55.750721 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:55.750732 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:55.750776 (+ 44us) server_negotiation.cc:407] Connection header received
0810 07:29:55.750958 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:55.750964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:55.751204 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:55.751410 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:55.752237 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.753206 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.754067 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.754654 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.754757 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:55.757795 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:55.757803 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:55.757819 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:55.757849 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:55.760002 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.760736 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.760742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.760744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.760894 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.761310 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.761313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.761315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.761908 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:55.763169 (+ 1261us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:55.763289 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 07:29:55.763440 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":311,"spinlock_wait_cycles":1408,"thread_start_us":127,"threads_started":1}
I20260810 07:29:55.765836 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.754240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:42589 (local address 127.0.0.1:34232)
0810 07:29:55.754436 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:55.754451 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:55.754525 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:55.754980 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:55.754987 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:55.755199 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:55.755305 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:55.755314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.756267 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.756274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:55.756846 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:55.756855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.757380 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.757387 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:55.757495 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:55.759055 (+ 1560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:55.759073 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:55.760731 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:55.763854 (+ 3123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:55.763857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:55.763859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:55.764099 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:55.764454 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:55.764457 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:55.764459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:55.764541 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:55.765310 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:55.765313 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:55.765435 (+ 122us) client_negotiation.cc:769] Sending connection context
0810 07:29:55.765542 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 07:29:55.765681 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260810 07:29:55.766430 17532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.754356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34232 (local address 127.12.157.252:42589)
0810 07:29:55.754699 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:55.754710 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:55.754725 (+ 15us) server_negotiation.cc:407] Connection header received
0810 07:29:55.754758 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:55.754760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:55.754810 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:55.755430 (+ 620us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:55.755519 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.756157 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.756982 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.757249 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.757367 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:55.761491 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:55.761497 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:55.761499 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:55.761520 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:55.763725 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.764229 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.764232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.764237 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.764348 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.764999 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.765003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.765004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.765192 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:55.765327 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:55.766127 (+ 800us) server_negotiation.cc:299] Negotiation successful
0810 07:29:55.766257 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":136,"threads_started":1}
I20260810 07:29:55.766417 17523 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33729 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:29:55.770035 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.758457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:42589 (local address 127.0.0.1:34244)
0810 07:29:55.758587 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:55.758605 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:55.758695 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:55.759409 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:55.759412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:55.759608 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:55.759730 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:55.759738 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.761127 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.761131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:55.762013 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:55.762028 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.762671 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.762675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:55.762760 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:55.763196 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:55.763211 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:55.764670 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:55.768433 (+ 3763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:55.768437 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:55.768439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:55.768699 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:55.769033 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:55.769036 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:55.769038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:55.769138 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:55.769564 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:55.769568 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:55.769664 (+ 96us) client_negotiation.cc:769] Sending connection context
0810 07:29:55.769750 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 07:29:55.769877 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 07:29:55.770512 17288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d39d9726d60641579229d0fb307e4239: Bootstrap starting.
I20260810 07:29:55.770701 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.758551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34244 (local address 127.12.157.252:42589)
0810 07:29:55.759119 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:55.759128 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:55.759143 (+ 15us) server_negotiation.cc:407] Connection header received
0810 07:29:55.759182 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:55.759185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:55.759234 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:55.759857 (+ 623us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:55.759964 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.761003 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.762162 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.762526 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.762991 (+ 465us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:55.764817 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:55.764821 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:55.764823 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:55.764846 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:55.768305 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.768819 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.768822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.768824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.768903 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.769246 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.769248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.769250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.769450 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:55.770288 (+ 838us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:55.770375 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 07:29:55.770490 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":373,"threads_started":1}
Aug 10 07:29:55 dist-test-slave-ffrd krb5kdc[17152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346995, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.252@KRBTEST.COM for kudu/127.12.157.253@KRBTEST.COM
I20260810 07:29:55.773782 17288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d39d9726d60641579229d0fb307e4239: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:55.774057 17412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca53018f8e8b4b51ae3a1e86dcce5286: Bootstrap starting.
Aug 10 07:29:55 dist-test-slave-ffrd krb5kdc[17152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.254@KRBTEST.COM
I20260810 07:29:55.775277 17288 log.cc:824] T 00000000000000000000000000000000 P d39d9726d60641579229d0fb307e4239: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:55.777159 17412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca53018f8e8b4b51ae3a1e86dcce5286: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:55.777665 17419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.767201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55612 (local address 127.12.157.253:33729)
0810 07:29:55.767527 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:55.767533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:55.767547 (+ 14us) server_negotiation.cc:407] Connection header received
0810 07:29:55.767583 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:55.767586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:55.767638 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:55.767718 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:55.768227 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.769066 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.770121 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.770472 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.770549 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:55.773361 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:55.773365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:55.773368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:55.773392 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:55.775102 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.776159 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.776163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.776164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.776251 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.776589 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.776592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.776594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.776852 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:55.776926 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:55.777374 (+ 448us) server_negotiation.cc:299] Negotiation successful
0810 07:29:55.777519 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238}
I20260810 07:29:55.777812 17288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d39d9726d60641579229d0fb307e4239: No bootstrap required, opened a new log
I20260810 07:29:55.778053 17412 log.cc:824] T 00000000000000000000000000000000 P ca53018f8e8b4b51ae3a1e86dcce5286: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:55.778213 17525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.767077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.253:33729 (local address 127.0.0.1:55612)
0810 07:29:55.767393 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:55.767408 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:55.767481 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:55.767748 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:55.767752 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:55.767941 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:55.768079 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:55.768086 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.769199 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.769206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:55.769987 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:55.769998 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.770703 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.770707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:55.770832 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:55.771339 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:55.771353 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:55.773204 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:55.775506 (+ 2302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:55.775510 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:55.775513 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:55.776042 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:55.776385 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:55.776388 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:55.776390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:55.776481 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:55.777043 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:55.777048 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:55.777164 (+ 116us) client_negotiation.cc:769] Sending connection context
0810 07:29:55.777884 (+ 720us) client_negotiation.cc:240] Negotiation successful
0810 07:29:55.778044 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229}
I20260810 07:29:55.779454 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.768027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58226 (local address 127.12.157.254:37121)
0810 07:29:55.768154 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:55.768159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:55.768459 (+ 300us) server_negotiation.cc:407] Connection header received
0810 07:29:55.768495 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:55.768498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:55.768536 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:55.768598 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:55.769489 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.770164 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.771437 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.771731 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.771849 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:55.774863 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:55.774867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:55.774870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:55.774900 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:55.776549 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.777087 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.777091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.777092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.777178 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.777566 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.777569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.777571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.778065 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:55.778531 (+ 466us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:55.779175 (+ 644us) server_negotiation.cc:299] Negotiation successful
0810 07:29:55.779309 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 07:29:55.780237 17412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca53018f8e8b4b51ae3a1e86dcce5286: No bootstrap required, opened a new log
I20260810 07:29:55.782248 17288 raft_consensus.cc:348] T 00000000000000000000000000000000 P d39d9726d60641579229d0fb307e4239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39d9726d60641579229d0fb307e4239" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37121 } } peers { permanent_uuid: "ca53018f8e8b4b51ae3a1e86dcce5286" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33729 } } peers { permanent_uuid: "2f606e1479be49d885704ca1d4ffabdd" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 42589 } }
I20260810 07:29:55.782437 17288 raft_consensus.cc:729] T 00000000000000000000000000000000 P d39d9726d60641579229d0fb307e4239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d39d9726d60641579229d0fb307e4239, State: Initialized, Role: FOLLOWER
I20260810 07:29:55.782696 17288 consensus_queue.cc:260] T 00000000000000000000000000000000 P d39d9726d60641579229d0fb307e4239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39d9726d60641579229d0fb307e4239" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37121 } } peers { permanent_uuid: "ca53018f8e8b4b51ae3a1e86dcce5286" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33729 } } peers { permanent_uuid: "2f606e1479be49d885704ca1d4ffabdd" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 42589 } }
I20260810 07:29:55.784068 17288 sys_catalog.cc:564] T 00000000000000000000000000000000 P d39d9726d60641579229d0fb307e4239 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:55.783960 17412 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca53018f8e8b4b51ae3a1e86dcce5286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39d9726d60641579229d0fb307e4239" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37121 } } peers { permanent_uuid: "ca53018f8e8b4b51ae3a1e86dcce5286" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33729 } } peers { permanent_uuid: "2f606e1479be49d885704ca1d4ffabdd" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 42589 } }
I20260810 07:29:55.784147 17412 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca53018f8e8b4b51ae3a1e86dcce5286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca53018f8e8b4b51ae3a1e86dcce5286, State: Initialized, Role: FOLLOWER
I20260810 07:29:55.784449 17412 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca53018f8e8b4b51ae3a1e86dcce5286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39d9726d60641579229d0fb307e4239" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37121 } } peers { permanent_uuid: "ca53018f8e8b4b51ae3a1e86dcce5286" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33729 } } peers { permanent_uuid: "2f606e1479be49d885704ca1d4ffabdd" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 42589 } }
I20260810 07:29:55.785260 17552 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca53018f8e8b4b51ae3a1e86dcce5286 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39d9726d60641579229d0fb307e4239" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37121 } } peers { permanent_uuid: "ca53018f8e8b4b51ae3a1e86dcce5286" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33729 } } peers { permanent_uuid: "2f606e1479be49d885704ca1d4ffabdd" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 42589 } } }
I20260810 07:29:55.785475 17552 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca53018f8e8b4b51ae3a1e86dcce5286 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:29:55.785514 17523 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.157.252" port: 42589 } has no permanent_uuid. Determining permanent_uuid...
I20260810 07:29:55.786052 17412 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca53018f8e8b4b51ae3a1e86dcce5286 [sys.catalog]: configured and running, proceeding with master startup.
Aug 10 07:29:55 dist-test-slave-ffrd krb5kdc[17152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.157.253@KRBTEST.COM
I20260810 07:29:55.788266 17546 sys_catalog.cc:455] T 00000000000000000000000000000000 P d39d9726d60641579229d0fb307e4239 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39d9726d60641579229d0fb307e4239" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37121 } } peers { permanent_uuid: "ca53018f8e8b4b51ae3a1e86dcce5286" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33729 } } peers { permanent_uuid: "2f606e1479be49d885704ca1d4ffabdd" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 42589 } } }
I20260810 07:29:55.791146 17546 sys_catalog.cc:458] T 00000000000000000000000000000000 P d39d9726d60641579229d0fb307e4239 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:29:55.791689 17419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.780842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55628 (local address 127.12.157.253:33729)
0810 07:29:55.780966 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:55.780970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:55.780984 (+ 14us) server_negotiation.cc:407] Connection header received
0810 07:29:55.781172 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:55.781175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:55.781220 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:55.781304 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:55.782057 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.782840 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.783853 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.784164 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.784278 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:55.787324 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:55.787330 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:55.787334 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:55.787369 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:55.789412 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.789934 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.789938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.789940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.790032 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.790337 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.790340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.790342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.790552 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:55.790782 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:55.791420 (+ 638us) server_negotiation.cc:299] Negotiation successful
0810 07:29:55.791541 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 07:29:55.791712 17579 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d39d9726d60641579229d0fb307e4239: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:29:55.791774 17579 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 10 07:29:55 dist-test-slave-ffrd krb5kdc[17152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786346995, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.157.252@KRBTEST.COM for kudu/127.12.157.252@KRBTEST.COM
I20260810 07:29:55.797251 12919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 07:29:55.798661 17525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.786075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.157.252:42589 (local address 127.0.0.1:34256)
0810 07:29:55.787956 (+ 1881us) negotiation.cc:107] Waiting for socket to connect
0810 07:29:55.787973 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 07:29:55.788064 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 07:29:55.788863 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 07:29:55.788866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 07:29:55.789103 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0810 07:29:55.789211 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:55.789221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.790163 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.790166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:55.791087 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 07:29:55.791101 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.792151 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.792154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 07:29:55.792266 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:55.792980 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 07:29:55.792999 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 07:29:55.794590 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 07:29:55.797050 (+ 2460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:55.797053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:55.797055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:55.797297 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:55.797570 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 07:29:55.797575 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 07:29:55.797578 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 07:29:55.797643 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 07:29:55.798132 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 07:29:55.798136 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 07:29:55.798229 (+ 93us) client_negotiation.cc:769] Sending connection context
0810 07:29:55.798351 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 07:29:55.798495 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1794}
I20260810 07:29:55.799130 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.786240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34256 (local address 127.12.157.252:42589)
0810 07:29:55.786383 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:55.786388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:55.788035 (+ 1647us) server_negotiation.cc:407] Connection header received
0810 07:29:55.788579 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:55.788582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:55.788636 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:55.788732 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:55.789347 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.790046 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.791646 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.791990 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.792105 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:55.794723 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:55.794728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:55.794730 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:55.794753 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:55.796915 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.797412 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.797418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.797421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.797490 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.797736 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.797739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.797742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.797989 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:55.798095 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:55.798866 (+ 771us) server_negotiation.cc:299] Negotiation successful
0810 07:29:55.798992 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 07:29:55.800004 17589 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ca53018f8e8b4b51ae3a1e86dcce5286: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:29:55.800097 17589 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:29:55.797180 17181 master_runner.cc:186] Error getting master registration for 127.12.157.252:42589: OK, instance_id { permanent_uuid: "2f606e1479be49d885704ca1d4ffabdd" instance_seqno: 1786346995736058 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 07:29:55.800937 17181 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 07:29:55.801446 17255 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d39d9726d60641579229d0fb307e4239"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58226
I20260810 07:29:55.801704 17255 raft_consensus.cc:482] T 00000000000000000000000000000000 P d39d9726d60641579229d0fb307e4239 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 07:29:55.801855 17255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d39d9726d60641579229d0fb307e4239 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:55.799058 17300 master_runner.cc:186] Error getting master registration for 127.12.157.252:42589: OK, instance_id { permanent_uuid: "2f606e1479be49d885704ca1d4ffabdd" instance_seqno: 1786346995736058 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 07:29:55.802582 17300 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 07:29:55.803162 17255 raft_consensus.cc:504] T 00000000000000000000000000000000 P d39d9726d60641579229d0fb307e4239 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39d9726d60641579229d0fb307e4239" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37121 } } peers { permanent_uuid: "ca53018f8e8b4b51ae3a1e86dcce5286" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33729 } } peers { permanent_uuid: "2f606e1479be49d885704ca1d4ffabdd" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 42589 } }
I20260810 07:29:55.804206 17255 leader_election.cc:288] T 00000000000000000000000000000000 P d39d9726d60641579229d0fb307e4239 [CANDIDATE]: Term 1 election: Requested vote from peers ca53018f8e8b4b51ae3a1e86dcce5286 (127.12.157.253:33729), 2f606e1479be49d885704ca1d4ffabdd (127.12.157.252:42589)
I20260810 07:29:55.804677 17372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d39d9726d60641579229d0fb307e4239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ca53018f8e8b4b51ae3a1e86dcce5286"
I20260810 07:29:55.804931 17372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca53018f8e8b4b51ae3a1e86dcce5286 [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:55.806406 17372 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ca53018f8e8b4b51ae3a1e86dcce5286 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d39d9726d60641579229d0fb307e4239 in term 1.
I20260810 07:29:55.806910 17213 leader_election.cc:302] T 00000000000000000000000000000000 P d39d9726d60641579229d0fb307e4239 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ca53018f8e8b4b51ae3a1e86dcce5286, d39d9726d60641579229d0fb307e4239; no voters:
I20260810 07:29:55.807263 17523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f606e1479be49d885704ca1d4ffabdd: Bootstrap starting.
I20260810 07:29:55.808554 17546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d39d9726d60641579229d0fb307e4239 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 07:29:55.808804 17546 raft_consensus.cc:686] T 00000000000000000000000000000000 P d39d9726d60641579229d0fb307e4239 [term 1 LEADER]: Becoming Leader. State: Replica: d39d9726d60641579229d0fb307e4239, State: Running, Role: LEADER
I20260810 07:29:55.809203 17546 consensus_queue.cc:237] T 00000000000000000000000000000000 P d39d9726d60641579229d0fb307e4239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39d9726d60641579229d0fb307e4239" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37121 } } peers { permanent_uuid: "ca53018f8e8b4b51ae3a1e86dcce5286" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33729 } } peers { permanent_uuid: "2f606e1479be49d885704ca1d4ffabdd" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 42589 } }
I20260810 07:29:55.810220 17523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f606e1479be49d885704ca1d4ffabdd: Neither blocks nor log segments found. Creating new log.
I20260810 07:29:55.811110 17523 log.cc:824] T 00000000000000000000000000000000 P 2f606e1479be49d885704ca1d4ffabdd: Log is configured to *not* fsync() on all Append() calls
I20260810 07:29:55.811110 17546 sys_catalog.cc:455] T 00000000000000000000000000000000 P d39d9726d60641579229d0fb307e4239 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d39d9726d60641579229d0fb307e4239. Latest consensus state: current_term: 1 leader_uuid: "d39d9726d60641579229d0fb307e4239" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39d9726d60641579229d0fb307e4239" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37121 } } peers { permanent_uuid: "ca53018f8e8b4b51ae3a1e86dcce5286" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33729 } } peers { permanent_uuid: "2f606e1479be49d885704ca1d4ffabdd" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 42589 } } }
I20260810 07:29:55.811265 17546 sys_catalog.cc:458] T 00000000000000000000000000000000 P d39d9726d60641579229d0fb307e4239 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:55.811650 17607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 07:29:55.812763 17607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 07:29:55.818828 17372 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ca53018f8e8b4b51ae3a1e86dcce5286 [term 1 FOLLOWER]: Refusing update from remote peer d39d9726d60641579229d0fb307e4239: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:29:55.819582 17603 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d39d9726d60641579229d0fb307e4239 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca53018f8e8b4b51ae3a1e86dcce5286" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:29:55.824319 17552 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca53018f8e8b4b51ae3a1e86dcce5286 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d39d9726d60641579229d0fb307e4239. Latest consensus state: current_term: 1 leader_uuid: "d39d9726d60641579229d0fb307e4239" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39d9726d60641579229d0fb307e4239" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37121 } } peers { permanent_uuid: "ca53018f8e8b4b51ae3a1e86dcce5286" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33729 } } peers { permanent_uuid: "2f606e1479be49d885704ca1d4ffabdd" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 42589 } } }
I20260810 07:29:55.824501 17552 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca53018f8e8b4b51ae3a1e86dcce5286 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:29:55.828351 17611 mvcc.cc:205] Tried to move back new op lower bound from 7316877294864330752 to 7316877294838185984. Current Snapshot: MvccSnapshot[applied={T|T < 7316877294864330752}]
I20260810 07:29:55.828572 17546 sys_catalog.cc:455] T 00000000000000000000000000000000 P d39d9726d60641579229d0fb307e4239 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d39d9726d60641579229d0fb307e4239" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39d9726d60641579229d0fb307e4239" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37121 } } peers { permanent_uuid: "ca53018f8e8b4b51ae3a1e86dcce5286" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33729 } } peers { permanent_uuid: "2f606e1479be49d885704ca1d4ffabdd" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 42589 } } }
I20260810 07:29:55.828712 17546 sys_catalog.cc:458] T 00000000000000000000000000000000 P d39d9726d60641579229d0fb307e4239 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:55.829741 17552 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca53018f8e8b4b51ae3a1e86dcce5286 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d39d9726d60641579229d0fb307e4239" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39d9726d60641579229d0fb307e4239" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37121 } } peers { permanent_uuid: "ca53018f8e8b4b51ae3a1e86dcce5286" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33729 } } peers { permanent_uuid: "2f606e1479be49d885704ca1d4ffabdd" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 42589 } } }
I20260810 07:29:55.829867 17552 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca53018f8e8b4b51ae3a1e86dcce5286 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:29:55.830586 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.815293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58232 (local address 127.12.157.254:37121)
0810 07:29:55.815413 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:55.815419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:55.818616 (+ 3197us) server_negotiation.cc:407] Connection header received
0810 07:29:55.818814 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:55.818818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:55.818880 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:55.818934 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:55.820035 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.820815 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.822057 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.822399 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.822516 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:55.825301 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:55.825306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:55.825308 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:55.825336 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:55.827118 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.828620 (+ 1502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.828625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.828629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.828720 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.829115 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.829118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.829121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.829354 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:55.829482 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:55.830322 (+ 840us) server_negotiation.cc:299] Negotiation successful
0810 07:29:55.830443 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 07:29:55.832414 17523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f606e1479be49d885704ca1d4ffabdd: No bootstrap required, opened a new log
I20260810 07:29:55.833614 17607 catalog_manager.cc:1347] Generated new cluster ID: a4d950f1b2c8418191cd81106802b0b0
I20260810 07:29:55.833675 17607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 07:29:55.838151 17523 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f606e1479be49d885704ca1d4ffabdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39d9726d60641579229d0fb307e4239" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37121 } } peers { permanent_uuid: "ca53018f8e8b4b51ae3a1e86dcce5286" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33729 } } peers { permanent_uuid: "2f606e1479be49d885704ca1d4ffabdd" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 42589 } }
I20260810 07:29:55.839252 17523 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f606e1479be49d885704ca1d4ffabdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f606e1479be49d885704ca1d4ffabdd, State: Initialized, Role: FOLLOWER
I20260810 07:29:55.839862 17523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f606e1479be49d885704ca1d4ffabdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39d9726d60641579229d0fb307e4239" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37121 } } peers { permanent_uuid: "ca53018f8e8b4b51ae3a1e86dcce5286" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33729 } } peers { permanent_uuid: "2f606e1479be49d885704ca1d4ffabdd" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 42589 } }
I20260810 07:29:55.843660 17480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d39d9726d60641579229d0fb307e4239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2f606e1479be49d885704ca1d4ffabdd"
I20260810 07:29:55.843950 17480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f606e1479be49d885704ca1d4ffabdd [term 0 FOLLOWER]: Advancing to term 1
I20260810 07:29:55.845448 17607 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 07:29:55.845716 17480 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2f606e1479be49d885704ca1d4ffabdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d39d9726d60641579229d0fb307e4239 in term 1.
I20260810 07:29:55.845840 17615 mvcc.cc:205] Tried to move back new op lower bound from 7316877294864330752 to 7316877294838185984. Current Snapshot: MvccSnapshot[applied={T|T < 7316877294864330752 or (T in {7316877294864330752})}]
I20260810 07:29:55.846352 17479 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2f606e1479be49d885704ca1d4ffabdd [term 1 FOLLOWER]: Refusing update from remote peer d39d9726d60641579229d0fb307e4239: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 07:29:55.847020 17523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f606e1479be49d885704ca1d4ffabdd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 07:29:55.847468 17607 catalog_manager.cc:1504] Loading token signing keys...
I20260810 07:29:55.849356 17603 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d39d9726d60641579229d0fb307e4239 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f606e1479be49d885704ca1d4ffabdd" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 42589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 07:29:55.851641 17419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.818135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55636 (local address 127.12.157.253:33729)
0810 07:29:55.818289 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:55.818295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:55.837252 (+ 18957us) server_negotiation.cc:407] Connection header received
0810 07:29:55.837554 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:55.837558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:55.837612 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:55.837665 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:55.839097 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.839897 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.841107 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.841428 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.841531 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:55.843540 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:55.843544 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:55.843548 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:55.843577 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:55.846847 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.848313 (+ 1466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.848317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.848319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.848499 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.848855 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.848859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.848861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.849146 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:55.849273 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:55.851336 (+ 2063us) server_negotiation.cc:299] Negotiation successful
0810 07:29:55.851481 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260810 07:29:55.855918 17637 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2f606e1479be49d885704ca1d4ffabdd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 07:29:55.856003 17637 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 07:29:55.853737 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:55.817704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34260 (local address 127.12.157.252:42589)
0810 07:29:55.817842 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:55.817847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:55.830217 (+ 12370us) server_negotiation.cc:407] Connection header received
0810 07:29:55.831849 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:55.831852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:55.831910 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:55.831975 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:55.832919 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.833655 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.834775 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:55.845156 (+ 10381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:55.845266 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:55.849409 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 07:29:55.849413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 07:29:55.849416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 07:29:55.849444 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 07:29:55.851212 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.852218 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.852221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.852223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.852310 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 07:29:55.852634 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 07:29:55.852637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 07:29:55.852639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 07:29:55.852878 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 07:29:55.853036 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:55.853467 (+ 431us) server_negotiation.cc:299] Negotiation successful
0810 07:29:55.853591 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 07:29:55.857008 17646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f606e1479be49d885704ca1d4ffabdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d39d9726d60641579229d0fb307e4239" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39d9726d60641579229d0fb307e4239" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37121 } } peers { permanent_uuid: "ca53018f8e8b4b51ae3a1e86dcce5286" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33729 } } peers { permanent_uuid: "2f606e1479be49d885704ca1d4ffabdd" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 42589 } } }
I20260810 07:29:55.857147 17646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f606e1479be49d885704ca1d4ffabdd [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:29:55.857252 17646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f606e1479be49d885704ca1d4ffabdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d39d9726d60641579229d0fb307e4239. Latest consensus state: current_term: 1 leader_uuid: "d39d9726d60641579229d0fb307e4239" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39d9726d60641579229d0fb307e4239" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37121 } } peers { permanent_uuid: "ca53018f8e8b4b51ae3a1e86dcce5286" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33729 } } peers { permanent_uuid: "2f606e1479be49d885704ca1d4ffabdd" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 42589 } } }
I20260810 07:29:55.857326 17646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f606e1479be49d885704ca1d4ffabdd [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:29:55.857399 17646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f606e1479be49d885704ca1d4ffabdd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d39d9726d60641579229d0fb307e4239" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39d9726d60641579229d0fb307e4239" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37121 } } peers { permanent_uuid: "ca53018f8e8b4b51ae3a1e86dcce5286" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33729 } } peers { permanent_uuid: "2f606e1479be49d885704ca1d4ffabdd" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 42589 } } }
I20260810 07:29:55.857453 17646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f606e1479be49d885704ca1d4ffabdd [sys.catalog]: This master's current role is: FOLLOWER
I20260810 07:29:55.861122 17641 mvcc.cc:205] Tried to move back new op lower bound from 7316877294966001664 to 7316877294838185984. Current Snapshot: MvccSnapshot[applied={T|T < 7316877294864330752}]
I20260810 07:29:55.861817 17546 sys_catalog.cc:455] T 00000000000000000000000000000000 P d39d9726d60641579229d0fb307e4239 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d39d9726d60641579229d0fb307e4239" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d39d9726d60641579229d0fb307e4239" member_type: VOTER last_known_addr { host: "127.12.157.254" port: 37121 } } peers { permanent_uuid: "ca53018f8e8b4b51ae3a1e86dcce5286" member_type: VOTER last_known_addr { host: "127.12.157.253" port: 33729 } } peers { permanent_uuid: "2f606e1479be49d885704ca1d4ffabdd" member_type: VOTER last_known_addr { host: "127.12.157.252" port: 42589 } } }
I20260810 07:29:55.861937 17546 sys_catalog.cc:458] T 00000000000000000000000000000000 P d39d9726d60641579229d0fb307e4239 [sys.catalog]: This master's current role is: LEADER
I20260810 07:29:55.863128 17607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d39d9726d60641579229d0fb307e4239: Generated new TSK 0
I20260810 07:29:55.863941 17607 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 07:29:55.901105 17291 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:55.897613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58244 (local address 127.12.157.254:37121)
0810 07:29:55.897745 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:55.897750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:55.899716 (+ 1966us) server_negotiation.cc:407] Connection header received
0810 07:29:55.900131 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:55.900136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:55.900192 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:55.900251 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:55.900925 (+ 674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58244: BlockingRecv error: recv got EOF from 127.0.0.1:58244 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 07:29:55.902462 17534 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:55.898409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34264 (local address 127.12.157.252:42589)
0810 07:29:55.898525 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:55.898529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:55.900041 (+ 1512us) server_negotiation.cc:407] Connection header received
0810 07:29:55.901694 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:55.901698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:55.901751 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:55.901802 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:55.902339 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34264: BlockingRecv error: recv got EOF from 127.0.0.1:34264 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 07:29:55.903623 17419 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:55.899074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55648 (local address 127.12.157.253:33729)
0810 07:29:55.899186 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:55.899190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:55.901510 (+ 2320us) server_negotiation.cc:407] Connection header received
0810 07:29:55.902816 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:55.902821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:55.902881 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:55.902944 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:55.903506 (+ 562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55648: BlockingRecv error: recv got EOF from 127.0.0.1:55648 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260810 07:29:55.908414 17419 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:55.906860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55658 (local address 127.12.157.253:33729)
0810 07:29:55.906990 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:55.906994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:55.907469 (+ 475us) server_negotiation.cc:407] Connection header received
0810 07:29:55.907686 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:55.907689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:55.907767 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:55.907818 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:55.908298 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55658: BlockingRecv error: recv got EOF from 127.0.0.1:55658 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260810 07:29:55.913228 17419 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:55.911740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55672 (local address 127.12.157.253:33729)
0810 07:29:55.911837 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:55.911841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:55.912351 (+ 510us) server_negotiation.cc:407] Connection header received
0810 07:29:55.912532 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:55.912535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:55.912580 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:55.912627 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:55.913138 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55672: BlockingRecv error: recv got EOF from 127.0.0.1:55672 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260810 07:29:55.918423 17419 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:55.917024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55686 (local address 127.12.157.253:33729)
0810 07:29:55.917114 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:55.917118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:55.917613 (+ 495us) server_negotiation.cc:407] Connection header received
0810 07:29:55.917783 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:55.917787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:55.917831 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:55.917879 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:55.918340 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55686: BlockingRecv error: recv got EOF from 127.0.0.1:55686 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260810 07:29:55.925648 17419 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:55.924167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55692 (local address 127.12.157.253:33729)
0810 07:29:55.924292 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:55.924296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:55.924788 (+ 492us) server_negotiation.cc:407] Connection header received
0810 07:29:55.924999 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:55.925002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:55.925047 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:55.925095 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:55.925564 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55692: BlockingRecv error: recv got EOF from 127.0.0.1:55692 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260810 07:29:55.937129 17419 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:55.935550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55708 (local address 127.12.157.253:33729)
0810 07:29:55.935662 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:55.935666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:55.936178 (+ 512us) server_negotiation.cc:407] Connection header received
0810 07:29:55.936380 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:55.936383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:55.936430 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:55.936478 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:55.937033 (+ 555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55708: BlockingRecv error: recv got EOF from 127.0.0.1:55708 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 07:29:55.957020 17419 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:55.955338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55718 (local address 127.12.157.253:33729)
0810 07:29:55.955461 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:55.955465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:55.956030 (+ 565us) server_negotiation.cc:407] Connection header received
0810 07:29:55.956227 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:55.956231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:55.956282 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:55.956346 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:55.956877 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55718: BlockingRecv error: recv got EOF from 127.0.0.1:55718 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260810 07:29:55.993522 17419 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:55.991604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55732 (local address 127.12.157.253:33729)
0810 07:29:55.991736 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:55.991741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:55.992374 (+ 633us) server_negotiation.cc:407] Connection header received
0810 07:29:55.992607 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:55.992611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:55.992671 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:55.992742 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:55.993404 (+ 662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55732: BlockingRecv error: recv got EOF from 127.0.0.1:55732 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260810 07:29:56.061225 17419 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:56.059900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55746 (local address 127.12.157.253:33729)
0810 07:29:56.060042 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:56.060047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:56.060288 (+ 241us) server_negotiation.cc:407] Connection header received
0810 07:29:56.060456 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:56.060460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:56.060516 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:56.060604 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:56.061108 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55746: BlockingRecv error: recv got EOF from 127.0.0.1:55746 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 07:29:56.192961 17419 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:56.191580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55760 (local address 127.12.157.253:33729)
0810 07:29:56.191762 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:56.191769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:56.191946 (+ 177us) server_negotiation.cc:407] Connection header received
0810 07:29:56.192152 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:56.192156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:56.192223 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:56.192323 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:56.192811 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55760: BlockingRecv error: recv got EOF from 127.0.0.1:55760 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260810 07:29:56.452517 17419 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 07:29:56.451246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55768 (local address 127.12.157.253:33729)
0810 07:29:56.451429 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:56.451433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:56.451450 (+ 17us) server_negotiation.cc:407] Connection header received
0810 07:29:56.451656 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:56.451660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:56.451722 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:56.451835 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 07:29:56.452399 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55768: BlockingRecv error: recv got EOF from 127.0.0.1:55768 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 07:29:56.801532 17589 catalog_manager.cc:1259] Loaded cluster ID: a4d950f1b2c8418191cd81106802b0b0
I20260810 07:29:56.801625 17589 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ca53018f8e8b4b51ae3a1e86dcce5286: loading cluster ID for follower catalog manager: success
I20260810 07:29:56.803279 17589 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ca53018f8e8b4b51ae3a1e86dcce5286: acquiring CA information for follower catalog manager: success
I20260810 07:29:56.804596 17589 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ca53018f8e8b4b51ae3a1e86dcce5286: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 07:29:56.857442 17637 catalog_manager.cc:1259] Loaded cluster ID: a4d950f1b2c8418191cd81106802b0b0
I20260810 07:29:56.857522 17637 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2f606e1479be49d885704ca1d4ffabdd: loading cluster ID for follower catalog manager: success
I20260810 07:29:56.859138 17637 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2f606e1479be49d885704ca1d4ffabdd: acquiring CA information for follower catalog manager: success
I20260810 07:29:56.860244 17637 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2f606e1479be49d885704ca1d4ffabdd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 07:29:56.970481 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:56.966811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55774 (local address 127.12.157.253:33729)
0810 07:29:56.967083 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:56.967086 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:56.967103 (+ 17us) server_negotiation.cc:407] Connection header received
0810 07:29:56.967206 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:56.967209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:56.967339 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:56.967475 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:29:56.967697 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:56.968221 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:56.969112 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:56.969338 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:56.969424 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:56.969689 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:29:56.970136 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:29:56.970270 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:56.970297 (+ 27us) server_negotiation.cc:299] Negotiation successful
0810 07:29:56.970359 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":104,"threads_started":1}
I20260810 07:29:56.977450 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:56.973724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34274 (local address 127.12.157.252:42589)
0810 07:29:56.973995 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:56.973999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:56.974017 (+ 18us) server_negotiation.cc:407] Connection header received
0810 07:29:56.974188 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:56.974190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:56.974304 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:56.974410 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:29:56.974636 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:56.975158 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:56.975960 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:56.976195 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:56.976288 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:56.976529 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:29:56.976934 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:29:56.977098 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:56.977183 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 07:29:56.977249 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":102,"threads_started":1}
I20260810 07:29:56.984887 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:56.980992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34278 (local address 127.12.157.252:42589)
0810 07:29:56.981118 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:56.981124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:56.981512 (+ 388us) server_negotiation.cc:407] Connection header received
0810 07:29:56.981676 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:56.981682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:56.981734 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:56.981795 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:29:56.982052 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:56.982801 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:56.983649 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:56.983995 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:56.984119 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:56.984429 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:29:56.984523 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:29:56.984597 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:56.984682 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 07:29:56.984745 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 07:29:56.985121 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:56.980565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55786 (local address 127.12.157.253:33729)
0810 07:29:56.980691 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:56.980704 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:56.980983 (+ 279us) server_negotiation.cc:407] Connection header received
0810 07:29:56.981172 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:56.981175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:56.981245 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:56.981296 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:29:56.981892 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:56.982645 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:56.983749 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:56.984093 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:56.984189 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:56.984581 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:29:56.984663 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:29:56.984732 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:56.984881 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 07:29:56.984973 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 07:29:56.985884 17785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 07:29:56.980560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58260 (local address 127.12.157.254:37121)
0810 07:29:56.980836 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0810 07:29:56.980846 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 07:29:56.980866 (+ 20us) server_negotiation.cc:407] Connection header received
0810 07:29:56.981132 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 07:29:56.981135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 07:29:56.981320 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 07:29:56.981458 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 07:29:56.981722 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:56.982394 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:56.983539 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 07:29:56.983894 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 07:29:56.984004 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 07:29:56.984904 (+ 900us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 07:29:56.985364 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 07:29:56.985458 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 07:29:56.985611 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 07:29:56.985682 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":76,"threads_started":1}
I20260810 07:29:56.989174 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 17181
I20260810 07:29:57.006597 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 17300
I20260810 07:29:57.021463 12919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasko6sLg5/build/debug/bin/kudu with pid 17421
2026-08-10T07:29:57Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3336 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6701 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (302051 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST